Reinforcement learning tutorial with TensorFlow

Reinforcement learning TensorFlow - action and training steps

Reinforcement learning has gained significant attention with the relatively recent success of DeepMind’s AlphaGo system defeating the world champion Go player. The AlphaGo system was trained in part by reinforcement learning on deep neural networks. This type of learning is a different aspect of machine learning from the classical supervised and unsupervised paradigms. In reinforcement learning using deep neural networks, the network reacts to environmental data (called the state) and controls the actions of an agent to attempt to maximize a reward. This process allows a network to learn to play games, such as Atari or other video games, or any other problem that can be recast as some form of game. In this tutorial, I’ll introduce the broad concepts of Q learning, a popular reinforcement learning paradigm, and I’ll show how to implement deep Q learning in TensorFlow. If you need to get up to speed in TensorFlow, check out my introductory tutorial.

Introduction to reinforcement learning

As stated above, reinforcement learning comprises of a few fundamental entities or concepts. They are: an environment which produces a state and reward, and an agent which performs actions in the given environment. This interaction can be seen in the diagram below:

Reinforcement learning with Python and Keras - Reinforcement learning environment

Reinforcement learning environment

The goal of the agent in such an environment is to examine the state and the reward information it receives, and choose an action which maximizes the reward feedback it receives.  The agent learns by repeated interaction with the environment, or, in other words, repeated playing of the game.

To be successful, the agent needs to:

  1. Learn the interaction between states, actions and subsequent rewards
  2. Determine which is the best action to choose given (1)

The implementation of (1) involves determining some set of values which can be used to inform (2), and (2) is called the action policy. One of the most common ways of implementing (1) and (2) using deep learning is via the Deep Q network and the epsilon-greedy policy. I’ll cover both of these concepts in the next two sections.

Q learning

Q learning is a value based method of supplying information to inform which action an agent should take. An initially intuitive idea of creating values upon which to base actions is to create a table which sums up the rewards of taking action in state s over multiple game plays. This could keep track of which moves are the most advantageous. For instance, let’s consider a simple game which has 3 states and two possible actions in each state – the rewards for this game can be represented in a table:

Reinforcement learning - simple state-action-reward table

Simple state-action-reward table

In the table above, you can see that for this simple game, when the agent is State 1 and takes Action 2, it will receive a reward of 10 but zero reward if it takes Action 1. In State 2, the situation is reversed and finally State 3 resembles State 1. If an agent randomly explored this game, and summed up which actions received the most reward in each of the three states (storing this information in an array, say), then it would basically learn the functional form of the table above.

In other words, if the agent simply chooses the action which it learnt had yielded the highest reward in the past (effectively learning some form of the table above) it would have learnt how to play the game successfully. Why do we need fancy concepts such as Q learning and neural networks then, when simply creating tables by summation is sufficient?

Deferred reward

Well, the first obvious answer is that the game above is clearly very simple, with only 3 states and 2 actions per state. Real games are significantly more complex. The other significant concept that is missing in the example above is the idea of deferred reward. To adequately play most realistic games, an agent needs to learn to be able to take actions which may not immediately lead to a reward, but may result in a large reward further down the track.

Consider another game, defined by the table below:

Reinforcement learning with TensorFlow - simple delayed reward value table

Simple delayed reward value table

In the game defined above, in all states, if Action 2 is taken, the agent moves back to State 1 i.e. it goes back to the beginning. In States 1 to 3, it also receives a reward of 5 when it does so. However, in all States 1 – 3, if Action 1 is taken, the agent moves forward to the next state, but doesn’t receive a reward until it reaches State 4 – at which point it receives a reward of 20. In other words, an agent is better off if it doesn’t take Action 2 to get an instantaneous reward of 5, but rather it should choose Action 1 consistently to progress through the states to get the reward of 20. The agent needs to be able to select actions which result in a delayed reward, if the delayed reward value is sufficiently large.

The Q learning rule

This allows us to define the Q learning rule. In deep Q learning, the neural network needs to take the current state, s, as a variable and return a Q value for each possible action, a, in that state – i.e. it needs to return $Q(s,a)$ for all s and a. This $Q(s,a)$ needs to be updated in training via the following rule:

$$Q(s,a) \leftarrow  Q(s,a) + \alpha [r + \gamma \max_{a’} Q(s’, a’) – Q(s,a)]$$

This updating rule needs a bit of unpacking. First, you can see that the new value of $Q(s,a)$ involves updating it’s current value by adding on some extra bits on the right hand side of the equation above. Moving left to right, ignore the $\alpha$ for a bit. We see inside the square brackets the first term is r which stands for the reward that is received for taking action a in state s. This is the immediate reward, no delayed gratification is involved yet.

The next term is the delayed reward calculation. First, we have the $\gamma$ value which discounts the delayed reward impact – it is always between 0 and 1. More on that in a second. The next term $\max_{a’} Q(s’, a’)$ is the maximum Q value possible in the next state. Let’s make that a bit clearer – the agent starts in state s, takes action a, ends up in state s’ and then the code determines the maximum Q value in state s’  i.e. $\max_{a’} Q(s’, a’)$.

So why is the value $\max_{a’} Q(s’, a’)$ considered? It is considered because it represents the maximum future reward coming to the agent if it takes action in state s. However, this value is discounted by $\gamma$ to take into account that it isn’t ideal for the agent to wait forever for a future reward – it is best for the agent to aim for the maximum award in the least period of time. Note that the value $Q(s’,a’)$ implicitly also holds the maximum discounted reward for the state after that, i.e. $Q(s”, a”)$ and likewise, it holds the discounted reward for the state $Q(s”’, a”’)$ and so on. This is how the agent can choose the action based on not just the immediate reward r, but also based on possible future discounted rewards.

The final components in the formula above are the $\alpha$ value, which is the learning rate during the updating, and finally the current state, $Q(s,a)$, which is subtracted from the square bracket sum. This is done to normalize the updating. Both $\alpha$ and the $Q(s,a)$ subtraction are not required to be explicitly defined in deep Q learning, as the neural network will take care of that during its optimized learning process. This process will be discussed in the next section.

Deep Q learning

Deep Q learning applies the Q learning updating rule during the training process. In other words, a neural network is created which takes the state as its input, and then the network is trained to output appropriate Q(s,a) values for each action in state s. The action of the agent can then be chosen by taking the action with the greatest Q(s,a) value (by taking an argmax of the output of the neural network). This can be seen in the first step of the diagram below:

Reinforcement learning TensorFlow - action and training steps

Action selecting and training steps – Deep Q learning

Once this step has been taken and an action has been selected, the agent can perform that action. The agent will then receive feedback on what reward is received by taking that action from that state. Now, the next step that we want to perform is to train the network according to the Q learning rule. This can be seen in the second part of the diagram above. The x input array for training the network is the state vector s, and the y output training sample is the Q(s,a) vector retrieved during the action selection step. However, one of the Q(s,a) values, corresponding to action a, is set to have a target of $r + \gamma Q(s’, a’)$ – this can be observed in the figure above.

By training the network in this way, the Q(s,a) output vector from the network will over time become better at informing the agent what action will be the best to select for its long term gain. There is a bit more to the story about action selection, however, which will be discussed in the next section.

The epsilon-greedy policy

In the explanation above, the action selection policy was simply the action which corresponded to the highest Q output from the neural network. However, this policy isn’t the most effective. Why is that? It is because, when the neural network is randomly initialized, it will be predisposed to select certain sub-optimal actions randomly. This may cause the agent to fall into sub-optimal behavior patterns without thoroughly exploring the game and action / reward space. As such, the agent won’t find the best strategies to play the game.

It is useful here to introduce two concepts – exploration and exploitation. At the beginning of an optimization problem, it is best to allow the problem space to be explored extensively in the hope of finding good local (or even global) minima. However, once the problem space has been adequately searched, it is now best for the optimization algorithm to focus on exploiting what it has found by converging on the best minima to arrive at a good solution.

Therefore, in reinforcement learning, it is best to allow some randomness in the action selection at the beginning of the training. This randomness is determined by the epsilon parameter. Essentially, a random number is drawn between 0 and 1, and if it is less than epsilon, then a random action is selection. If not, an action is selected based on the output of the neural network. The epsilon variable usually starts somewhere close to 1, and is slowly decayed to somewhere around 0 during training. This allows a large exploration of the game at the beginning, but then the decay of the epsilon value allows the network to zero in on a good solution.

We’re almost at the point where we can check out the game that will be used in this example, and begin to build our deep Q network. However, there is just one final important point to consider.

Batching in reinforcement learning

If a deep Q network is trained at each step in the game i.e. after each action is performed and the reward collected, there is a strong risk of over-fitting in the network. This is because game play is highly correlated i.e. if the game starts from the same place and the agent performs the same actions, there will likely be similar results each time (not exactly the same though, because of randomness in some games). Therefore, after each action it is a good idea to add all the data about the state, reward, action and the new state into some sort of memory. This memory can then be randomly sampled in batches to avoid the risk of over-fitting.

The network can therefore still be trained after each step if you desire (or less frequently, it’s up to the developer), but it is extracting the training data not from the agent’s ordered steps through the game, but rather a randomized memory of previous steps and outcomes that the agent has experienced. You’ll be able to see how this works in the code below.

We are now ready to examine the game/environment that we will develop our network to learn.

The Mountain Car Environment and Open AI Gym

In this reinforcement learning tutorial, the deep Q network that will be created will be trained on the Mountain Car environment/game. This can be accessed through the open source reinforcement learning library called Open AI Gym. A screen capture from the rendered game can be observed below:

Reinforcement learning TensorFlow - Mountain Car game

Mountain Car game

The object of this game is to get the car to go up the right-side hill to get to the flag. There’s one problem however, the car doesn’t have enough power to motor all the way up the hill. Instead, the car / agent needs to learn that it must motor up one hill for a bit, then accelerate down the hill and back up the other side, and repeat until it builds up enough momentum to make it to the top of the hill.

As stated above, Open AI Gym is an open source reinforcement learning package that allows developers to interact easily with games such as the Mountain Car environment. You can find details about the Mountain Car environment here. Basically, the environment is represented by a two-element state vector, detailed below:

Reinforcement learning - TensorFlow state vector

Mountain Car state vector

As can be observed, the agent’s state is represented by the car’s position and velocity. The goal/flag is sitting at a position = 0.5. The actions available to the agent are shown below:

Reinforcement learning TensorFlow - mountain car actions

Mountain Car actions

As can be observed, there are three actions available to the agent – accelerate to the left, right and no acceleration.

In the game’s default arrangement, for each time step where the car’s position is <0.5, it receives a reward of -1, up to a maximum of 200 time steps. So the incentive for the agent is to get the car’s position to >0.5 as soon as possible, after which the game ends. This will minimize the negative reward, which is the aim of the game.

However, in this default arrangement, it will take a significant period of time of random exploration before the car stumbles across the positive feedback of getting to the flag. As such, to speed things up a bit, in this example we’ll alter the reward structure to:

  • Position > 0.1, r += 10
  • Position > 0.25 r += 20
  • Position > 0.5 r += 100

This new reward structure gives the agent better positive feedback when it starts learning how to ascend the hill on the right hand side toward the flag. The position of 0.1 is just over half way up the right-hand hill.

Ok, so now you know the environment, let’s write some code!

Reinforcement learning in TensorFlow

In this reinforcement learning implementation in TensorFlow, I’m going to split the code up into three main classes, these classes are:

  • Model: This class holds the TensorFlow operations and model definitions
  • Memory: This class is where the memory of the actions, rewards and states are stored and retrieved from
  • GameRunner: This class is the main training and agent control class

As stated before, I’ll be assuming some prior knowledge of TensorFlow here. If you’re not up to speed your welcome to wing it. Otherwise check out my TensorFlow tutorial. All the code for this tutorial can be found on this site’s Github repository.

I’ll go through each of the classes in turn in the sub-sections below.

The Model class

class Model:
    def __init__(self, num_states, num_actions, batch_size):
        self._num_states = num_states
        self._num_actions = num_actions
        self._batch_size = batch_size
        # define the placeholders
        self._states = None
        self._actions = None
        # the output operations
        self._logits = None
        self._optimizer = None
        self._var_init = None
        # now setup the model
        self._define_model()

    def _define_model(self):
        self._states = tf.placeholder(shape=[None, self._num_states], dtype=tf.float32)
        self._q_s_a = tf.placeholder(shape=[None, self._num_actions], dtype=tf.float32)
        # create a couple of fully connected hidden layers
        fc1 = tf.layers.dense(self._states, 50, activation=tf.nn.relu)
        fc2 = tf.layers.dense(fc1, 50, activation=tf.nn.relu)
        self._logits = tf.layers.dense(fc2, self._num_actions)
        loss = tf.losses.mean_squared_error(self._q_s_a, self._logits)
        self._optimizer = tf.train.AdamOptimizer().minimize(loss)
        self._var_init = tf.global_variables_initializer()

The first function within the class is of course the initialization function. All you need to pass into the Model definition is the number of states of the environment (2 in this game), the number of possible actions (3 in this game) and the batch size. The function simply sets up a few internal variables and operations, some of which are exposed as public properties later in the class definition. At the end of the initialization, the second method displayed above _define_model() is called. This method sets up the model structure and the main operations.

First, two placeholders are created _states and _q_s_a – these hold the state data and the $Q(s,a)$ training data respectively. The first dimension of these placeholders is set to None, so that it will automatically adapt when a batch of training data is fed into the model and also when single predictions from the model are required. The next lines create two fully connected layers fc1 and fc2 using the handy TensorFlow layers module. These hidden layers have 50 nodes each, and they are activated using the ReLU activation function (if you want to know more about the ReLU, check out my vanishing gradient and ReLU tutorial).

The next layer is the output layer _logits – this is another fully connected or dense layer, but with no activation supplied. When no activation function is supplied to the dense layer API in TensorFlow, it defaults to a ‘linear’ activation i.e. no activation. This is what we want, as we want the network to learn continuous $Q(s,a)$ values across all possible real numbers.

Next comes the loss – this isn’t a classification problem, so a good loss to use is simply a mean squared error loss. The next line specifies the optimizer – in this example, we’ll just use the generic Adam optimizer. Finally, the TensorFlow boiler plate global variable initializer operation is assigned to _var_init.

So far so good. Next, some methods of the Model class are created to perform prediction and training:

    def predict_one(self, state, sess):
        return sess.run(self._logits, feed_dict={self._states:
                                                     state.reshape(1, self.num_states)})

    def predict_batch(self, states, sess):
        return sess.run(self._logits, feed_dict={self._states: states})

    def train_batch(self, sess, x_batch, y_batch):
        sess.run(self._optimizer, feed_dict={self._states: x_batch, self._q_s_a: y_batch})

The first method predict_one simply returns the output of the network (i.e. by calling the _logits operation) with an input of a single state. Note the reshaping operation that is used to ensure that the data has a size (1, num_states). This is called whenever action selection by the agent is required. The next method, predict_batch predicts a whole batch of outputs when given a whole bunch of input states – this is used to perform batch evaluation of $Q(s,a)$ and $Q(s’,a’)$ values for training. Finally, there is a method called train_batch which takes a batch training step of the network.

That’s the Model class, now it is time to consider the Memory class.

The Memory class

The next class to consider in the code is the Memory class – this class stores all the results of the action of the agent in the game, and also handles the retrieval. These can be used to batch train the network.

class Memory:
    def __init__(self, max_memory):
        self._max_memory = max_memory
        self._samples = []

    def add_sample(self, sample):
        self._samples.append(sample)
        if len(self._samples) > self._max_memory:
            self._samples.pop(0)

    def sample(self, no_samples):
        if no_samples > len(self._samples):
            return random.sample(self._samples, len(self._samples))
        else:
            return random.sample(self._samples, no_samples)

First, when the Memory class is initialized, it is necessary to supply a maximum memory argument – this will control the maximum number of (state, action, reward, next_state) tuples the _samples list can hold. The bigger the better, as it ensures better random mixing of the samples, but you have to make sure you don’t run into memory errors.

The first method, add_sample takes an individual (state, action, reward, next_state) tuple and appends it to the _samples list. After this, a check is made – if the number of samples is now larger than the allowable memory size, the first element in _samples is removed using the Python .pop() list functionality.

The final method, sample returns a random selection of no_samples in length. However, if the no_samples argument is larger than the actual memory, whatever is available in the memory is returned.

The final class is called GameRunner.

The GameRunner class

The GameRunner class in this example is where all the model dynamics, agent action and training is organised.

class GameRunner:
    def __init__(self, sess, model, env, memory, max_eps, min_eps,
                 decay, render=True):
        self._sess = sess
        self._env = env
        self._model = model
        self._memory = memory
        self._render = render
        self._max_eps = max_eps
        self._min_eps = min_eps
        self._decay = decay
        self._eps = self._max_eps
        self._steps = 0
        self._reward_store = []
        self._max_x_store = []

In the GameRunner initialization, some internal variables are created. Note, it takes as first argument a TensorFlow session object, then a neural network Model, an Open AI gym environment and a Memory class instance. The next arguments max_eps and min_eps dictate the maximum and minimum epsilon values respectively – during training the actual $\epsilon$ will decay from the maximum to the minimum based on the following argument decay. Finally, render is a boolean which determines whether the game environment is rendered to the screen.

The next method is run():

    def run(self):
        state = self._env.reset()
        tot_reward = 0
        max_x = -100
        while True:
            if self._render:
                self._env.render()

            action = self._choose_action(state)
            next_state, reward, done, info = self._env.step(action)
            if next_state[0] >= 0.1:
                reward += 10
            elif next_state[0] >= 0.25:
                reward += 20
            elif next_state[0] >= 0.5:
                reward += 100

            if next_state[0] > max_x:
                max_x = next_state[0]
            # is the game complete? If so, set the next state to
            # None for storage sake
            if done:
                next_state = None

            self._memory.add_sample((state, action, reward, next_state))
            self._replay()

            # exponentially decay the eps value
            self._steps += 1
            self._eps = MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) \
                                      * math.exp(-LAMBDA * self._steps)

            # move the agent to the next state and accumulate the reward
            state = next_state
            tot_reward += reward

            # if the game is done, break the loop
            if done:
                self._reward_store.append(tot_reward)
                self._max_x_store.append(max_x)
                break

        print("Step {}, Total reward: {}, Eps: {}".format(self._steps, tot_reward, self._eps))

We’ll go through each step in the code above. First, the environment is reset by calling the Open AI Gym command .reset(). Then an infinite loop is entered into – this will be exited by calling a break command. If the boolean _render is True, then the output of the game will be shown on the screen. The action of the agent is determined by calling the internal method _choose_action(state) – this will discussed later. Next, the agent takes action by calling the Open AI Gym command step(action). This command returns a tuple containing the new state of the agent, the reward received by taking action, a done boolean indicating whether the game has finished, and an information object (we won’t using info in this example).

The next step in the code is where there are some manual adjustments to the Mountain Car reward system. If you recall, earlier I mentioned that in order to speed up the training of the network, it was useful to add some more reward steps the closer the car got to the goal (rather than the default reward which was only received when the car reached the goal/flag). The maximum x value achieved in the given episode is also tracked and this will be stored once the game is complete.

The next step is a check to see if the game has completed i.e. done == True – this will occur after 200 turns. If it has completed, we want to set the next_state to None. This will be picked up during the training / replay step of the class, and the state will be set to an array of zeros whenever next_state is equal to None.

After this, the data about the agent is stored in the memory class – i.e.its original state, its chosen action, the reward it received for that action and finally the next_state of the agent. After this takes place, the training / replay step of the deep Q network is run – this step will be discussed more below. At this point the epsilon value is also exponentially decayed. Finally, the agent’s state is moved to next_state, the total reward during the game is accumulated, and there is some printing and breaking of the loop and storing of relevant variables if the game is complete.

The next part of the GameRunner class is the agent action selection method:

    def _choose_action(self, state):
        if random.random() < self._eps:
            return random.randint(0, self._model.num_actions - 1)
        else:
            return np.argmax(self._model.predict_one(state, self._sess))

This method executes our epsilon greedy + Q policy. In the first case, if a random number is less than the _eps value, then the returned action will simply be an action chosen at random from the set of possible actions. Otherwise, the action will be chosen based on an argmax of the output from the neural network. Recall that _predict_one from the model will take a single state as input, then output $Q(s,a)$ values for each of the possible actions available – the action with the highest $Q(s,a)$ value is that action with the highest expected current + future discounted reward.

The final method within the GameRunner class is the _replay method, where the batching and training takes place:

    def _replay(self):
        batch = self._memory.sample(self._model.batch_size)
        states = np.array([val[0] for val in batch])
        next_states = np.array([(np.zeros(self._model.num_states)
                                 if val[3] is None else val[3]) for val in batch])
        # predict Q(s,a) given the batch of states
        q_s_a = self._model.predict_batch(states, self._sess)
        # predict Q(s',a') - so that we can do gamma * max(Q(s'a')) below
        q_s_a_d = self._model.predict_batch(next_states, self._sess)
        # setup training arrays
        x = np.zeros((len(batch), self._model.num_states))
        y = np.zeros((len(batch), self._model.num_actions))
        for i, b in enumerate(batch):
            state, action, reward, next_state = b[0], b[1], b[2], b[3]
            # get the current q values for all actions in state
            current_q = q_s_a[i]
            # update the q value for action
            if next_state is None:
                # in this case, the game completed after action, so there is no max Q(s',a')
                # prediction possible
                current_q[action] = reward
            else:
                current_q[action] = reward + GAMMA * np.amax(q_s_a_d[i])
            x[i] = state
            y[i] = current_q
        self._model.train_batch(self._sess, x, y)

The first step in the _replay method is to retrieve a randomized batch of data from memory. Next, we want to setup our batch state variables so that we can:

  1. For each state, produce baseline $Q(s,a)$ values – one of which will be given a target of $r + \gamma \max_{a’} Q(s’, a’)$
  2.  For each next_state, predict $Q(s’,a’)$ from the model, as required in (1)

Now, if you recall, each sample in memory has the form of a tuple: state, action, reward, next_state which was extracted from the game play. To setup a batch of initial states, then, we simply use Python list comprehension to extract the first tuple value from each sample in the batch. Likewise, we do the same for the fourth value in the tuple to extract the next_state value for each sample in the batch. Note that whenever the next_state corresponds to a case where the game finished (i.e. next_state is None) the next state value is replaced by a vector of zeros corresponding in size to the number of states in the game.

Next, the batch of $Q(s, a)$ and $Q(s’,a’)$ values are extracted from the model from states and next_states respectively. The and y training arrays are then created, but initially filled with zeros. After this, a loop is entered into to accumulate the and y values on which to train the model. Within this loop, we extract the memory values from the batch, then set a variable designating the Q values for the current state. If the next_state value is actually zero, there is no discounted future rewards to add, so the current_q corresponding to action is set a target of the reward only. Alternatively, if there is a valid next_state, then the current_q corresponding to action is set a target of the reward plus the discounted future reward i.e. $max_{a’} Q(s’, a’)$.

The state and current_q are then loaded into the and values for the given batch, until the batch data is completely extracted. Then the network is trained by calling _train_batch() on the model.

That completes the review of the main classes within the TensorFlow reinforcement learning example. All that is left is to setup the classes and enter the training loop.

The main function

The code below sets up the environment and the classes, and runs multiple games to perform the learning:

if __name__ == "__main__":
    env_name = 'MountainCar-v0'
    env = gym.make(env_name)

    num_states = env.env.observation_space.shape[0]
    num_actions = env.env.action_space.n

    model = Model(num_states, num_actions, BATCH_SIZE)
    mem = Memory(50000)

    with tf.Session() as sess:
        sess.run(model.var_init)
        gr = GameRunner(sess, model, env, mem, MAX_EPSILON, MIN_EPSILON,
                        LAMBDA)
        num_episodes = 300
        cnt = 0
        while cnt < num_episodes:
            if cnt % 10 == 0:
                print('Episode {} of {}'.format(cnt+1, num_episodes))
            gr.run()
            cnt += 1
        plt.plot(gr.reward_store)
        plt.show()
        plt.close("all")
        plt.plot(gr.max_x_store)
        plt.show()

In the first couple of lines, we create an Open AI Gym Mountain Car environment. Next, the number of states and actions are extracted from the environment object itself.

The network model and memory objects are then created – in this case, we’re using a batch size of 50 and a total number of samples in the memory of 50,000.

The TensorFlow session object is created, along with the variable initialization – then the GameRunner class is created. The number of episodes of the Mountain Car game which will be run in this training example is 300. For each of these episodes, we run the game by using the GameRunner run() method.

After all the episodes are run, some plotting is performed on the total reward for each episode, and the maximum x-axis value the cart reaches in the game (remembering that the goal is at x = 0.5). These plots can be observed below:

Mountain Car rewards - from the TensorFlow reinforcement learning example

The Mountain Car rewards from the TensorFlow reinforcement learning example

As can be observed, the network starts out controlling the agent rather poorly, while it is exploring the environment and accumulating memory. However once it starts to receive positive rewards by ascending the right-hand hill, the rewards rapidly increase.

Mountain Car maximum X - from the TensorFlow reinforcement learning example

The Mountain Car maximum x values from the TensorFlow reinforcement learning example

As can be observed above, while there is some volatility, the network learns that the best rewards are achieved by reaching the top of the right-hand hill and, towards the end of the training, consistently controls the car/agent to reach there.

This reinforcement learning tutorial in TensorFlow has shown you:

  1. The basics of Q learning
  2. The epsilon greed action selection policy
  3.  The importance of batching in training deep Q reinforcement learning networks, and
  4. How to implement a deep Q reinforcement learning network in TensorFlow

I hope it has been instructive – keep an eye out for future tutorials in reinforcement learning where more complicated games and techniques will be reviewed.

639 thoughts on “Reinforcement learning tutorial with TensorFlow”

  1. Andy, Great Tutorial!! Although I had to make modifications to make it work. That would be great to have the whole code including some parameters. Looking forward to more tutorials in RL!

  2. We are a group of volunteers and starting a new scheme
    in our community. Your website offered us with valuable info
    to work on. You’ve done a formidable job and our entire community will be grateful to
    you.

  3. Hey! This is my first visit to your blog! We are a collection of volunteers and starting a new project
    in a community in the same niche. Your blog provided us useful
    information to work on. You have done a marvellous job!

    Here is my website :: http://keonicbdgummy.ampedpages.com/Keoni-CBD-Take-Care-Of-Your-Body-Naturally–31821496; http://keonicbdgummy.ampedpages.com/Keoni-CBD-Take-Care-Of-Your-Body-Naturally–31821496,

  4. Good site! I truly love how it is simple on my eyes and the data are well
    written. I am wondering how I could be notified when a
    new post has been made. I’ve subscribed to your RSS which must do the trick!
    Have a great day!

    my homepage: 23.95.102.216

  5. With havin so much written content do you ever run into any problems of plagorism or copyright violation? My website has a lot of unique content
    I’ve either authored myself or outsourced but it appears
    a lot of it is popping it up all over the web without my permission. Do you
    know any ways to help prevent content from being ripped off?
    I’d definitely appreciate it.

    my website :: https://sites.google.com/view/keonicbdgummy/home (Clemmie)

  6. Thanks for a marvelous posting! I definitely enjoyed
    reading it, you’re a great author. I will make sure
    to bookmark your blog and may come back down the road.
    I want to encourage continue your great job, have a nice holiday weekend!

    Visit my web blog; organic foods

  7. Hey there! I know this is sort of off-topic however I had
    to ask. Does running a well-established website like yours require a massive amount work?

    I’m completely new to running a blog however I do write in my diary every day.
    I’d like to start a blog so I will be able to share my personal experience and feelings online.

    Please let me know if you have any suggestions or tips for brand
    new aspiring bloggers. Thankyou!

    Also visit my blog :: getting treatment

  8. Undeniably consider that that you stated. Your favorite justification seemed to be on the net the simplest factor
    to take note of. I say to you, I certainly get irked even as folks think about worries that they just
    don’t realize about. You controlled to hit the nail upon the top and outlined
    out the entire thing without having side-effects , other people
    could take a signal. Will likely be again to get more.
    Thank you

    Here is my webpage: Circadiyin; Doretha,

  9. eu adoro quando me deparo com artigos tão bem escritos quanto o que pude encontrar aqui neste blog .

    Vou continuar visitando esta página já que gostei muito dos seus textos
    .

  10. I really love your site.. Pleasant colors & theme.

    Did you build this amazing site yourself? Please reply back as I?m wanting to create my own site
    and would like to find out where you got this from or just what
    the theme is named. Kudos!

    my web site … Nouvee Skin Cream (Mammie)

  11. I’m really enjoying the design and layout of your site. It’s
    a very easy on the eyes which makes it much more enjoyable for me to
    come here and visit more often. Did you hire out a
    developer to create your theme? Exceptional work!

    Have a look at my webpage :: Pinnacle Science Testo Boost –
    http://www.tumblr.com

  12. Awesome site you have here but I was wanting
    to know if you knew of any user discussion forums that cover the
    same topics talked about in this article? I’d really like to be a part of online community where I can get feed-back
    from other knowledgeable individuals that share the same interest.
    If you have any suggestions, please let me know.
    Thanks a lot!

    Feel free to surf to my homepage Eulah

  13. I do not even know how I finished up right here, but I assumed this submit was once great.
    I don’t recognise who you might be but certainly you are going to a well-known blogger when you
    are not already 😉 Cheers!

    Visit my page Circadiyin – Amelia

  14. I loved as much as you’ll receive carried out right here.
    The sketch is attractive, your authored subject matter stylish.
    nonetheless, you command get bought an edginess over that you wish
    be delivering the following. unwell unquestionably
    come further formerly again as exactly the same nearly very
    often inside case you shield this increase.

    Also visit my homepage carb nite pdf

  15. After exploring a number of the articles on your site, I truly like your
    way of writing a blog. I bookmarked it to my bookmark website list and will be checking back in the
    near future. Take a look at my website too and tell me what you think.

    Here is my webpage Straight Gains XL [justpaste.it]

  16. Thank you for all your labor on this website. My mum takes pleasure in making time for investigations and
    it’s easy to understand why. My partner and
    i hear all relating to the lively ways you offer simple solutions
    through this web blog and as well as recommend contribution from the others on this content and
    our girl has been understanding a whole lot.
    Have fun with the rest of the year. Your performing a superb job.

    Feel free to surf to my web-site; Evonne

  17. Greate pieces. Keep writing such kind of information on your page.
    Im really impressed by your site.[X-N-E-W-L-I-N-S-P-I-N-X]Hey there, You’ve done an excellent
    job. I will certainly digg it and for my part recommend to my friends.

    I’m sure they’ll be benefited from this web site.

    Stop by my page … health foods

  18. Hey! I just wanted to ask if you ever have any issues with
    hackers? My last blog (wordpress) was hacked and I ended up losing several weeks of hard work due
    to no back up. Do you have any solutions to protect against hackers?

    my website … Diaetoxil

  19. Unquestionably believe that which you said. Your favorite justification appeared to be on the net
    the easiest thing to be aware of. I say to you, I definitely get annoyed while people consider worries that they just don’t know about.
    You managed to hit the nail upon the top and also defined out the whole thing without having side
    effect , people could take a signal. Will likely be back
    to get more. Thanks

    Here is my web blog – Straight Gains XL (https://docs.google.com/presentation/d/e/2PACX-1vQuzSH-uPrD9Yc78wIbfWB3pAnOF74Ip1Is-U7Ierm7rQeHw-f2_w6wLwZ3azX2OVShZniwM1BEpfaw/pub?start=false&loop=false&delayms=3000)

  20. Thanks for your marvelous posting! I genuinely enjoyed reading it, you happen to be
    a great author.I will be sure to bookmark your blog and will eventually come
    back very soon. I want to encourage that you continue your great work, have
    a nice weekend!

    Take a look at my web site; Pinnacle Science Testo Boost (groups.google.com)

  21. I know this if off topic but I’m looking into starting my own weblog and was
    curious what all is needed to get setup? I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very internet savvy so I’m not 100% certain. Any suggestions or advice would be greatly appreciated.

    Cheers

    Feel free to visit my web site – yizhangbang.net

  22. Hi! I understand this is kind of off-topic but I had to
    ask. Does operating a well-established website like yours take a large amount of
    work? I’m brand new to blogging however I do write in my diary every day.
    I’d like to start a blog so I will be able to share my own experience and views online.
    Please let me know if you have any kind of suggestions or tips for new aspiring blog owners.
    Thankyou!

    Visit my web site; growing weed

  23. hey there and thank you for your information ? I have certainly picked up anything new from right here.
    I did however expertise several technical points using this web site, as I
    experienced to reload the site many times previous to I could get it to
    load properly. I had been wondering if your web host
    is OK? Not that I am complaining, but sluggish loading instances
    times will often affect your placement in google
    and could damage your high-quality score if advertising and
    marketing with Adwords. Well I’m adding this RSS
    to my email and can look out for much more of your respective fascinating
    content. Ensure that you update this again very soon.

    Feel free to surf to my web-site: Boyce

  24. hello there and thank you for your information – I have
    certainly picked up something new from right here.
    I did however expertise a few technical issues using this site,
    since I experienced to reload the site lots of times
    previous to I could get it to load properly.
    I had been wondering if your hosting is OK?
    Not that I am complaining, but sluggish loading instances times will very frequently affect
    your placement in google and can damage your high-quality score if ads and marketing with Adwords.
    Anyway I’m adding this RSS to my email and can look out for
    much more of your respective interesting content.
    Ensure that you update this again very soon..

    Feel free to visit my web blog Quick Trim Keto – Nida

  25. Thanks , I’ve just been searching for information approximately this
    topic for a long time and yours is the best I have found out so far.
    However, what in regards to the bottom line? Are you certain in regards
    to the source?

    Feel free to surf to my page; Pinnacle Science Testo Boost, Keeley,

  26. Undeniably believe that which you stated. Your favourite reason appeared to be at the
    net the easiest thing to consider of. I say to you, I definitely get irked whilst people think about
    concerns that they plainly do not recognise about.
    You controlled to hit the nail upon the top and also defined out the entire thing
    with no need side-effects , other people can take a signal.
    Will probably be again to get more. Thanks

    Feel free to visit my blog – Circadiyin

  27. I was studying some of your content on this internet site and I conceive this internet site is real informative!
    Continue posting.

    Feel free to surf to my web-site: Quick Trim Keto – Lela

  28. I really love your website.. Great colors & theme. Did you develop this amazing site yourself?
    Please reply back as I?m hoping to create my very own site
    and would love to learn where you got this from or what the theme is called.
    Many thanks!

    Here is my website :: Pinnacle Science Testo Boost (Jordan)

  29. I have been surfing on-line more than three hours nowadays,
    yet I by no means discovered any interesting
    article like yours. It is pretty value sufficient for me.
    In my opinion, if all website owners and bloggers made good
    content material as you probably did, the internet
    might be a lot more helpful than ever before.

    Here is my web site: Diaetoxil – https://justpaste.it/5q96d

  30. Howdy! I know this is kinda off topic however , I’d
    figured I’d ask. Would you be interested in exchanging links or maybe guest writing a blog post
    or vice-versa? My blog addresses a lot of the
    same subjects as yours and I think we could greatly benefit from each other.
    If you might be interested feel free to shoot me an e-mail.

    I look forward to hearing from you! Wonderful blog by the way!

    Here is my web site seeds require long

  31. The very next time I read a blog, Hopefully it won’t disappoint me as much
    as this one. After all, Yes, it was my choice to
    read, however I actually believed you would probably have something useful to say.
    All I hear is a bunch of complaining about something you could
    possibly fix if you weren’t too busy seeking attention.

    my blog … casualvalueinvestor.com

  32. Hi there! I just wanted to ask if you ever have any trouble with hackers?
    My last blog (wordpress) was hacked and I ended up
    losing a few months of hard work due to no back up.
    Do you have any methods to prevent hackers?

    Review my page: Green Roads CBD (Zoila)

  33. Unquestionably believe that which you said. Your favorite reason seemed to
    be on the net the simplest thing to be aware of. I say to you, I
    definitely get irked while people think about worries that
    they just do not know about. You managed to hit
    the nail upon the top and defined out the whole
    thing without having side-effects , people can take a signal.

    Will likely be back to get more. Thanks

    Feel free to visit my blog; Straight Gains XL (https://kit.co/isaacforth/straight-gains-xl)

  34. Hey There. I found your blog using msn. This is an extremely well written article.
    I will make sure to bookmark it and return to read more of your useful
    information. Thanks for the post. I will certainly comeback.

    my website – Straight Gains XL (Trisha)

  35. I’ve been exploring for a little for any high-quality articles or blog posts on this sort of house
    . Exploring in Yahoo I finally stumbled upon this web site.

    Reading this info So i am glad to show that I’ve a very excellent
    uncanny feeling I came upon exactly what I needed. I most undoubtedly will make
    sure to do not forget this site and provides it a look
    on a constant basis.

    Here is my homepage :: Nouvee Skin Cream – Myrna,

  36. Having read this I believed it was extremely enlightening.
    I appreciate you spending some time and effort to put this informative article together.
    I once again find myself spending way too much time both reading and commenting.
    But so what, it was still worth it!

    Look into my web blog … lower belly

  37. We wish to thank you just as before for the stunning ideas you gave
    Jeremy when preparing a post-graduate research plus, most importantly, with regard to providing every one of the ideas in one blog post.
    If we had been aware of your blog a year ago, we might have been kept
    from the useless measures we were choosing. Thank you very much.

    Feel free to surf to my website … Straight Gains
    XL (Doyle)

  38. Wonderful blog! I found it while browsing on Yahoo News.

    Do you have any suggestions on how to get listed
    in Yahoo News? I’ve been trying for a while but I never seem to
    get there! Many thanks

    Look at my website; Quick Trim Keto (Charmain)

  39. I truly love your blog.. Very nice colors & theme.
    Did you make this web site yourself? Please reply
    back as I?m attempting to create my own personal blog and would like to learn where
    you got this from or just what the theme is called. Thanks!

    my webpage: Quick Trim Keto – Jacklyn

  40. I’m impressed, I have to admit. Rarely do I encounter a blog that’s both
    equally educative and amusing, and let me tell you, you’ve
    hit the nail on the head. The issue is something not enough folks are speaking intelligently about.
    Now i’m very happy I found this in my search for something relating to this.

    Here is my web page – Tacoma Farms CBD (docs.google.com)

  41. Can I simply just say what a relief to uncover someone that genuinely knows what they are
    talking about on the internet. You certainly realize how to bring a problem to light and
    make it important. More and more people ought to check this out and understand this side of your story.
    I can’t believe you are not more popular since you certainly have the
    gift.

    My web site fish oil

  42. I precisely wanted to thank you very much again. I am
    not sure the things that I would’ve made to happen without those basics discussed by you
    directly on such a area of interest. It was before an absolute fearsome setting for me,
    nevertheless witnessing your professional approach you resolved that made me to
    leap with contentment. I’m grateful for the information and
    as well , pray you realize what a great job you happen to be getting into training people using a blog.

    I am sure you haven’t met all of us.

    Feel free to surf to my homepage – http://bbs.inhe365.com/home.php?mod=space&uid=547011&do=profile&from=space

  43. We wish to thank you again for the stunning ideas you offered Jeremy when preparing her own post-graduate research and, most importantly, with regard
    to providing every one of the ideas in one blog post. If we had known of your web site a year ago,
    we’d have been rescued from the unnecessary measures we
    were taking. Thanks to you.

    Feel free to visit my page: http://159.203.199.234

  44. This is very attention-grabbing, You are an overly professional blogger.
    I have joined your rss feed and look forward to
    in search of more of your magnificent post. Additionally, I’ve
    shared your website in my social networks

    Here is my blog … healthy diet

  45. I’ve been exploring for a little for any high quality articles or weblog posts in this sort cause of hair loss in women area .
    Exploring in Yahoo I ultimately stumbled upon this
    website. Reading this info So i am glad to show that I’ve an incredibly
    good uncanny feeling I came upon just what I
    needed. I most indubitably will make certain to do not fail
    to remember this site and provides it a look regularly.

  46. This is the perfect site for anybody who wishes to understand this topic.
    You understand a whole lot its almost tough to argue with you (not that I personally will need to?HaHa).
    You definitely put a fresh spin on a topic that’s been discussed for decades.
    Great stuff, just great!

    Feel free to visit my web-site … amazing weight loss

  47. Its like you read my mind! You appear to know so much about this, like you wrote the
    book in it or something. I think that you could do with a few pics how to lose weight drive the message
    home a little bit, but instead of that, this is excellent blog.

    A fantastic read. I will definitely be back.

  48. hey there and thank you for your info ? I have
    definitely picked up anything new from right here. I did however expertise some
    technical points using this web site, since I experienced to reload the website lots of times previous
    to I could get it to load properly. I had been wondering if your web hosting is OK?
    Not that I am complaining, but sluggish loading instances times will often affect your placement in google and can damage your high-quality score if advertising and marketing with Adwords.

    Well I am adding this RSS how to lose weight my email and can look out for much more of your
    respective interesting content. Ensure that you update this
    again soon.

  49. Every weekend i used to pay a visit this web site, because i wish for enjoyment, for the reason that this
    this web page conations really pleasant funny material too.

    Also visit my webpage … Winona

  50. Please let me know if you’re looking for a article writer for your site.
    You have some really great articles and I believe I would be a good asset.

    If you ever want to take some of the load off, I’d absolutely love to write
    some content for your blog in exchange for a link back to mine.
    Please blast me an email if interested. Many thanks!

    Look at my site: forum.mamamj.ru

  51. Hello There. I found your weblog the use of msn. This is a very well written article.
    I will make sure to bookmark it and return to read extra of your helpful information. Thank you
    for the post. I will certainly comeback.

    Here is my web blog :: Anastasia

  52. you are really a excellent webmaster. The site loading pace is incredible.
    It kind of feels that you are doing any distinctive
    trick. Moreover, The contents are masterpiece. you’ve performed a excellent job in this subject!

    Also visit my web-site oil swishing

  53. I seldom drop responses, but i did some searching and wound up
    here Reinforcement learning tutorial with TensorFlow –
    Adventures in Machine Learning. And I actually do have a few questions
    for you if it’s allright. Could it be simply me or does it seem like some of these responses appear as if they are written by brain dead people?
    😛 And, if you are posting on additional places,
    I would like to follow anything fresh you have to post.
    Would you make a list of all of your communal pages
    like your linkedin profile, Facebook page or twitter
    feed?

    my website … eliminate yeast infection

  54. I cherished as much as you’ll obtain performed right here.
    The comic strip is tasteful, your authored material stylish.
    nonetheless, you command get bought an shakiness over that you want
    be turning in the following. unwell unquestionably come further formerly again since exactly the
    similar just about very frequently within case you defend this increase.

    Here is my homepage; cycling diet

  55. You could definitely see your skills in the paintings you write.

    The sector hopes for even more passionate writers
    such as you who are not afraid to say how they believe.
    At all times go after your heart.

    Look into my homepage :: Straight Gains XL – Marlon,

  56. hey there and thank you for your info ? I have certainly picked up anything
    new from right here. I did however expertise a few technical
    points using this site, since I experienced to reload the web site a lot of times previous to I could get it to load correctly.

    I had been wondering if your hosting is OK? Not that I’m complaining,
    but sluggish loading instances times will often affect
    your placement in google and could damage your high-quality
    score if ads and marketing with Adwords. Well I am adding this RSS to my
    email and can look out for much more of your respective interesting content.
    Ensure that you update this again soon.

    Feel free to visit my homepage improving sex

  57. Hey there this is somewhat of off topic but I was wanting to know if blogs use WYSIWYG editors or if you have
    to manually code with HTML. I’m starting a blog soon but have no coding
    knowledge so I wanted to get advice from someone
    with experience. Any help would be greatly appreciated!

    Feel free to visit my web-site :: anti aging solution

  58. First off I would like to say excellent blog!

    I had a quick question that I’d like to ask if you do not mind.
    I was interested to find out how you center yourself and clear your thoughts prior to writing.
    I’ve had a tough time clearing my thoughts in getting my ideas out there.
    I do enjoy writing but it just seems like the first 10 to 15 minutes are generally
    wasted simply just trying to figure out how to begin. Any suggestions or tips?
    Many thanks!

  59. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You obviously know what youre talking about, why waste your intelligence on just posting
    videos to your site when you could be giving us something informative to read?

    Check out my blog post; zicd.com

  60. Unquestionably consider that which you said. Your favourite reason seemed
    to be at the web the simplest factor to have in mind of.
    I say to you, I certainly get annoyed whilst folks think
    about issues that they just don’t recognise about.
    You controlled to hit the nail upon the top and also outlined out the entire thing without having side-effects , folks can take a signal.
    Will probably be again to get more. Thanks

    Also visit my website agrocase.ru

  61. Whats up very nice web site!! Man .. Excellent ..
    Amazing .. I will bookmark your site and take the feeds additionally…I am happy
    to seek out a lot of useful info right here within the post,
    we’d like work out more strategies in this regard,
    thank you for sharing.

    Look at my blog post … Circadiyin

  62. Please let me know if you’re looking for a author for your weblog.

    You have some really good articles and I believe I would be a good asset.
    If you ever want to take some of the load off, I’d love to write some articles for
    your blog in exchange diets for health
    a link back to mine. Please shoot me an email if interested.
    Thanks!

  63. I like the valuable info you provide in your articles.
    I will bookmark your blog and take a look at again right here regularly.
    I’m quite certain I will be told many new stuff right right here!
    Good luck for the next!

  64. My coder is trying to persuade me to move to .net from
    PHP. I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using Movable-type
    on numerous websites for about a year and am concerned about switching to another platform.
    I have heard fantastic things about blogengine.net.
    Is there a way I can transfer all my wordpress posts into it?

    Any kind of help would be really appreciated!

    Feel free to visit my webpage; drug abuse treatment

  65. Hello There. I found your blog using msn. This is an extremely well written article.
    I’ll make sure to bookmark it and come back to
    read more of your useful info. Thanks for
    the post. I will definitely comeback.

    Also visit my website … atolyesi.net

  66. Please let me know if you’re looking for a article author
    for your weblog. You have some really good articles and
    I feel I would be a good asset. If you ever want to take some of the load off, I’d really like to write
    some articles for your blog in exchange for a link back to mine.
    Please shoot me an e-mail if interested. Regards!

    my webpage :: omega 3 rich foods (thaipurchase.com)

  67. First of all I want to say terrific blog! I had a quick
    question which I’d like to ask if you do not mind. I was curious to find out
    how you center yourself and clear your thoughts before writing.
    I have had a tough time clearing my mind in getting
    my thoughts out. I do take pleasure in writing however it just seems like
    the first 10 to 15 minutes tend to be lost simply just trying to figure
    out how to begin. Any ideas or tips? Thanks!

    Here is my web site bbs.yunweishidai.com

  68. Hi, Neat post. There is a problem with your web site in internet explorer, might check
    this? IE still is the market leader and a big section of people will leave out
    your excellent writing because of this problem.

    Also visit my webpage … hatched seeds

  69. Hi there! I could have sworn I’ve been to this website before but after going through many of the articles I realized
    it’s new to me. Anyhow, I’m definitely delighted I discovered it and I’ll be book-marking it and checking back often!

    my homepage: Noella

  70. I have been exploring for a little bit for any high quality
    articles or weblog posts on this sort of area . Exploring in Yahoo I ultimately stumbled upon this site.

    Reading this information So i am satisfied to convey
    that I have an incredibly just right uncanny feeling I
    found out exactly what I needed. I such a lot unquestionably will make certain to do not disregard
    this web site and give it a look regularly.

    My web-site: cycling diet

  71. Do you have a spam problem on this site; I also am a blogger,
    and I was wondering your situation; many of us have developed some nice practices and
    we are looking to swap techniques with other folks,
    please shoot me an e-mail if interested.

    Check out my blog – good knife set

  72. I would like to point out my appreciation for your kindness giving support to people who have the need for assistance with your issue.
    Your real commitment to passing the message around had become particularly informative and have truly enabled men and women much like me to realize their goals.
    The warm and friendly instruction denotes a lot a person like me and even more to my peers.
    Thank you; from all of us.

    My site – cannabis license maybe

  73. Thank you, I’ve just been looking for information approximately this topic for ages and yours is the greatest I’ve came upon so far.

    But, what in regards to the bottom line? Are you certain about the source?

    my website – Magdalena

  74. I was wondering if you ever considered changing the structure of your
    website? Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so
    people could connect with it better. Youve got an awful lot of text for
    only having 1 or 2 images. Maybe you could space it out
    better?

    My page seeds require long

  75. Hey terrific blog! Does running a blog such as this require a massive amount work?

    I have no understanding of programming but I was hoping to start my
    own blog in the near future. Anyhow, should you have any
    suggestions or tips for new blog owners please share.
    I know this is off subject however I simply had to ask.
    Thanks!

    My blog post … http://www.aniene.net/modules.php?name=Your_Account&op=userinfo&username=VanderpoolJamika

  76. My wife and i felt quite joyful when Louis could deal with his survey from your precious recommendations he acquired from
    your weblog. It is now and again perplexing to simply find yourself
    offering concepts that the others could have been trying to sell.
    We see we now have you to be grateful to for that.
    The type of explanations you’ve made, the straightforward web site menu, the friendships your site help engender – it’s got mostly unbelievable, and it’s really leading our son in addition to the family imagine that
    that subject is excellent, weight loss and fasting that
    is exceptionally pressing. Thanks for all the pieces!

  77. What i don’t understood is in reality how you are now not really a lot more well-favored than you
    may be now. You are so intelligent. You understand therefore significantly when it
    comes to this topic, produced me in my opinion imagine it from so many various angles.
    Its like women and men are not fascinated except it
    is something to accomplish with Girl gaga!

    Your personal stuffs excellent. Always deal with it up!

    My web page: term treatment process

  78. I?m amazed, I have to admit. Rarely do I encounter a blog that?s equally educative
    and engaging, and without a doubt, you have hit the nail on the head.

    The issue is something that too few men and women are speaking intelligently about.
    Now i’m very happy I stumbled across this in my search for something regarding this.

    Here is my web blog anti aging solution

  79. Hi there very cool site!! Man .. Beautiful .. Amazing ..
    I’ll bookmark your website and take the feeds additionally?

    I’m happy to find a lot of helpful information right here
    within the publish, we want work out more techniques on this regard, thank you for sharing.
    . . . . .

    my webpage; Nicolas

  80. Please let me know if you’re looking for a article
    author for your weblog. You have some really great posts and
    I think I would be a good asset. If you ever want to take some of the load off, I’d
    really like to write some content for your blog in exchange for a link back
    to mine. Please send me an e-mail if interested.
    Thanks!

    my web-site … atkins diet weight loss protein based diet revolution diet plan surrounding atkins diet diet book

  81. Good ? I should certainly pronounce, impressed with your website.

    I had no trouble navigating through all the tabs and related information ended
    up being truly easy to do to access. I recently
    found what I hoped for before you know it at all. Reasonably unusual.

    Is likely to appreciate it for those who add forums or anything, site theme .
    a tones way for your client to communicate.
    Excellent task.

    Here is my page – weed indoors

  82. I just wanted to thank you a lot more for your amazing
    web page you have produced here. It really is full of ideas for those who are truly interested in this specific subject, primarily this very post.
    Your all so sweet and thoughtful of others plus reading your site posts is a wonderful delight
    if you ask me. And what generous present! Ben and
    I will certainly have pleasure making use of your tips in what we have
    to do in the near future. Our checklist is a mile long which means
    that your tips will be put to fine use.

    my web site sylvbuster.free.fr

  83. Howdy I am so thrilled I found your blog page, I really found you by
    mistake, while I was searching on Askjeeve for something else,
    Anyways I am here now and would just like to say many thanks for a tremendous post and
    a all round exciting blog (I also love the theme/design), I
    don’t have time to read it all at the minute but I have bookmarked it and also added your RSS feeds, so when I have time I will be back to read a
    lot more, Please do keep up the fantastic work.

    Feel free to surf to my webpage; carb nite

  84. Hey! I realize this is somewhat off-topic but I needed to ask.
    Does managing a well-established website such as
    yours take a massive amount work? I’m completely new to blogging however I do write in my diary daily.
    I’d like to start a blog so I can share my personal experience and feelings online.

    Please let me know if you have any recommendations
    or tips for new aspiring blog owners. Appreciate it!

    Also visit my web blog; best weight loss pills

  85. Hey There. I found your blog using msn. This is an extremely well written article.
    I’ll make sure to bookmark it and come back to read more of your useful info.
    Thanks for the post. I’ll certainly return.

    Feel free to surf to my web blog – fad diets

  86. I think this is one of the most significant info for me.
    And i’m glad reading your article. But wanna remark
    on some general things, The web site style is ideal,
    the articles is really nice : D. Good job, cheers

    My website :: cure eczema

  87. Very nice post. I just stumbled upon your weblog and wished to say that I’ve really enjoyed surfing around
    your blog posts. After all I will be subscribing to your
    feed and I hope you write again very soon!

    my web page – sciatica sleep (Rudy)

  88. What’s Happening i am new to this, I stumbled upon this I have
    found It absolutely helpful and it has aided me out loads.
    I hope to contribute & help other customers like its aided me.
    Great job.

    Also visit my web-site :: Larry

  89. Thank you for sharing excellent informations. Your web site is so
    cool. I am impressed by the details that you have on this blog.

    It reveals how nicely you understand this subject.
    Bookmarked this web page, will come back for
    more articles. You, my pal, ROCK! I found
    just the info I already searched everywhere and simply
    could not come across. What a perfect website.

    My site effective skin care tips

  90. Howdy just wanted to give you a quick heads up. The text in your
    post seem to be running off the screen in Opera.
    I’m not sure if this is a format issue or something to do with
    browser compatibility but I figured I’d post to let you know.
    The layout look great though! Hope you get the problem
    fixed soon. Cheers

    my web blog – http://www.hltkd.tw

  91. Very good website you have here but I was wondering if you knew of any user discussion forums that cover the same
    topics talked about here? I’d really like to be a part of
    group where I can get suggestions from other experienced people
    that share the same interest. If you have any recommendations, please let
    me know. Many thanks!

    My page – internet marketing

  92. I think everything posted made a ton of sense. However, what
    about this? suppose you added a little content?
    I am not suggesting your content isn’t good., however suppose you
    added a title that grabbed people’s attention? I mean Reinforcement learning tutorial with TensorFlow –
    Adventures in Machine Learning is a little plain. You could peek at Yahoo’s front page
    and see how they write post titles to grab viewers to open the
    links. You might add a related video or a related picture or two to get people interested about
    everything’ve written. Just my opinion, it would make your posts a little bit
    more interesting.

    My web site – stop fat gain

  93. My spouse and I absolutely love your blog and find almost all of your post’s
    to be exactly I’m looking for. can you offer guest writers to write content in your case?

    I wouldn’t mind publishing a post or elaborating on a lot of the subjects you write concerning here.
    Again, awesome weblog!

    Here is my web blog … low carb

  94. Fantastic blog! Do you have any hints for aspiring
    writers? I’m planning to start my own blog soon but I’m a little lost
    on everything. Would you advise starting with a free platform like WordPress or go for a
    paid option? There are so many choices out there that I’m
    totally overwhelmed .. Any suggestions? Kudos!

    Also visit my blog: best anti-aging skin care

  95. I blog often and I genuinely thank you for your information. This
    article has truly peaked my interest. I am going to take a note of your blog and keep checking for new information about once a week.
    I subscribed to your Feed too.

    Feel free to visit my web blog; top ten skin

  96. Wonderful items from you, man. I have consider your stuff prior
    to and you’re just extremely excellent. I actually like what you have acquired here, certainly like what you’re saying and
    the way in which during which you say it. You’re making it entertaining and you still care
    for to keep it wise. I can not wait how to burn fat read
    much more from you. That is actually a tremendous site.

  97. I’ve been surfing online more than 2 hours today, yet I never found any interesting article like yours.
    It’s pretty worth enough for me. Personally, if all website
    owners and bloggers made good content as you did, the internet will be a lot more useful
    than ever before.

    Check out my website http://www.a913.vip

  98. Having read this I thought it was rather informative. I appreciate you finding the time
    and energy to put this information together. I
    once again find myself personally spending a significant amount of time both reading and
    commenting. But so what, it was still worth it!

    Visit my web site :: cure eczema

  99. What i don’t realize is actually how you are now not really a lot more well-appreciated than you may be now.
    You’re very intelligent. You recognize therefore significantly in relation to this matter, produced
    me in my view believe it from numerous various angles.
    Its like women and men are not interested unless it is something
    to accomplish with Lady gaga! Your personal stuffs excellent.

    All the time deal with it up!

    my web page … seeds require

  100. Hey there! I know this is sort of off-topic but I had to ask.

    Does managing a well-established website such as yours require a massive amount
    work? I’m brand new to blogging however I do write in my diary
    on a daily basis. I’d like to start a blog so I will be able to share my personal experience
    and thoughts online. Please let me know if you have any kind of ideas or tips for new aspiring bloggers.
    Appreciate it!

    my web site: https://zwiazek-zawodowy-opiekunek.pl

  101. The next time I read a blog, Hopefully it won’t disappoint me as much as this one.
    I mean, I know it was my choice to read through, but I truly thought you’d have something interesting to talk about.
    All I hear is a bunch of crying about something you could fix if you weren’t
    too busy searching for attention.

    Stop by my blog anxiety pill available

  102. hello!,I like your writing so a lot! proportion we keep in touch extra about your post on AOL?
    I require an expert on this space to unravel my problem.

    Maybe that’s you! Looking ahead to look you.

    my web site :: seo tips

  103. Unquestionably imagine that which you said. Your favourite reason seemed to be at the internet the easiest factor to have
    in mind of. I say to you, I definitely get annoyed at the same time as other people think about issues
    that they just don’t recognise about. You managed to hit the nail upon the top
    as smartly as defined out the entire thing with no need side-effects , folks could take a signal.
    Will probably be back to get more. Thanks!

    Review my web blog; genital hair removal (http://www.fles.hlc.edu.tw/userinfo.php?uid=1416105)

  104. I simply wished to thank you very much once again. I’m not certain what I could possibly
    have taken care of without the type of information documented by you on my subject matter.
    It became a real frightening difficulty in my opinion, however , considering a new specialised technique you resolved the issue made me to
    weep over fulfillment. I will be happier for your information and even expect you really know what a great job you were
    doing teaching the mediocre ones using a web site.
    I am certain you have never come across all of us.

    Here is my page: foro.mecanicasa.es

  105. Hey there just wanted to give you a quick heads up.
    The text in your post seem to be running off the screen in Firefox.
    I’m not sure if this is a format issue or something to do
    with internet browser compatibility but I thought I’d post to let you know.
    The style and design look great though! Hope you get the issue resolved soon. Kudos

  106. Great – I should definitely pronounce, impressed with
    your web site. I had no trouble navigating through all
    the tabs and related info ended up being truly easy to
    do to access. I recently found what I hoped for before you know it
    in the least. Reasonably unusual. Is likely to appreciate it for those
    who add forums or anything, website theme . a tones way for your client to communicate.
    Excellent task.

    Here is my web page – diet pill (http://www.fles.hlc.edu.tw)

  107. I simply wanted to thank you yet again for your amazing web page you have built here.
    It truly is full of ideas for those who are really interested in this particular subject, especially this very post.
    Your all so sweet and also thoughtful of others and reading the blog posts is
    a wonderful delight in my opinion. And that of a generous
    reward! Ben and I will certainly have excitement making use of your recommendations
    in what we should instead do in a month’s time. Our checklist is
    a mile long so your tips will be put to fine use.

    Feel free to visit my site :: find easy diets

  108. Having read this I thought it was rather enlightening. I appreciate you finding the time and energy to put this article together.
    I once again find myself personally spending way too
    much time both reading and commenting. But so what, it was still worth it!

    Look into my web page – seed sprouts

  109. We’re a gaggle of volunteers and opening a brand new scheme in our community.
    Your website provided us with helpful information to work
    on. You’ve performed an impressive job and our entire neighborhood shall be thankful to you.

    Take a look at my web-site: folding knives

  110. I carry on listening to the news bulletin speak about receiving free online grant
    applications so I have been looking around for the top site to
    get one. Could you advise me please, where could i find
    some?

    my site: Nicolas

  111. I’m not sure the place you are getting your information, however great topic.
    I must spend some time studying more or figuring out more.
    Thank you for great information I was on the lookout for this info for
    my mission.

    Also visit my webpage :: diabetic diet

  112. Excellent website you have here but I was wondering if
    you knew of any discussion boards that cover the same topics talked about here?
    I’d really like to be a part of community where I can get
    responses from other knowledgeable people that share the same interest.
    If you have any suggestions, please let me know.
    Cheers!

    Look at my web site: belly fat supplements

  113. Can I simply say what a relief to discover an individual who really
    understands what they are talking about over the internet.
    You definitely realize how to bring a problem to light
    and make it important. More and more people should look at
    this and understand this side of the story. I can’t believe you aren’t
    more popular since you surely possess the gift.

    my web site … cannabis seeds

  114. Can I simply just say what a comfort to uncover somebody
    who actually understands what they’re discussing online. You
    actually know how to bring an issue to light and make it important.

    More and more people really need to look at this and understand this
    side of your story. It’s surprising you are not more popular because you most
    certainly possess the gift.

    my page: http://www.volleyball.com.hk

  115. I love your blog.. very nice colors & theme. Did you create this
    website yourself or did you hire someone to do it
    for you? Plz reply as I’m looking to design my own blog and would like to find out where u got this from.

    thank you

    Here is my site :: hatched seeds

  116. Hey there I am so excited I found your blog page, I really found you by error, while I was browsing on Digg for
    something else, Anyways I am here now and would just like to say thanks a
    lot for a marvelous post and a all round enjoyable
    blog (I also love the theme/design), I don’t have time to browse it all at
    the moment but I have book-marked it and also added in your RSS feeds, so when I have time I will be back to read a great deal
    more, Please do keep up the great job.

    Also visit my page http://www.dumankayahifit.com