Expert system is a software system which tries to mimic a human expert. A Rules system is a specific type of Expert System, where the knowledge is systematically fed into the system in the form of rules. The idea is that we can get a human domain expert to write down their understanding about how a problem should be solved. Then the software takes over and solves any instances of the problem that is presented to it.
 
These rules which encode the knowledge or expertise are typically in a simple enough format that a human expert should be able to write them or review them.  

A common example used to demonstrate Expert systems is a Blocks world. In a blocks world, there are different Lego type blocks which are placed on a surface. The blocks are of different sizes and colours. The initial configuration of these blocks are then fed into the Expert System as facts. The system is then given a puzzle to solve involving these blocks.

Another common use case of Rules System is Airline reward points calculation, calculating discount for customers on e-commerce platforms and medical diagnosis.

Lets say, in the blocks world, we are given the following facts:

Then, we are given a puzzle or a rule like find a stack of two blocks to the left of a red block? A Rules engine will be able to output the specific block numbers which satisfy this condition or output that none can.

RETE (Latin for "network") is a popular algorithm which is used to implement a Rules Engine. RETE algorithm takes into two types of inputs. Firstly, the rules which encode the facts describing the current problem at hand called working memory and the expert knowledge, called productions rules. It then checks the facts and outputs which of the production rules are currently satisfied. An excellent exposition of RETE Algorithm is here.

A fact or Working memory element (often written WME) consists of 3 parts - subject, predicate and object. A fact cannot have a variable. For example, the first fact in the above example can be written as: 

A Rule or Production rule has a set of conditions and a set of actions. A rule can have variables. A rule is considered satisfied, if all its conditions are fulfilled with the variables being assigned consistently. Finally, if all the conditions of a particular rule is satisfied, then its actions are executed. For example, the production rule in the above example can be written as:
x on y
y left-of z
z color red

Where x, y and z are variables. It is RETE Algorithm's responsibility to find a set of blocks from B1, B2, B3 and B4 which can be assigned to x, y and z such that the rule is satisfied.

In this example, there is only one production rule. However, in a real world case, there can be multiple. 

Internally, RETE algorithm builds a data-structure, based on the conditions in the rules. This data structure, called a network has 2 parts - Alpha and Beta as well as join nodes. Whenever a new fact or WME is presented, it is evaluated using this network and final output is calculated. RETE stores intermediate results, so re-evaluation after a small change in WMEs is fast.

RETE can handle a large number of facts and rules. Also, if the facts keep changing, that is, they are added, modified or deleted, it can quickly update the results. Typically, the upper limit is 100K rules depending on hardware.