Reinforcement learning tutorial using Python and Keras

Reinforcement learning - Python and Keras - NChain environment

In this post, I’m going to introduce the concept of reinforcement learning, and show you how to build an autonomous agent that can successfully play a simple game. Reinforcement learning is an active and interesting area of machine learning research, and has been spurred on by recent successes such as the AlphaGo system, which has convincingly beat the best human players in the world. This occurred in a game that was thought too difficult for machines to learn. In this tutorial, I’ll first detail some background theory while dealing with a toy game in the Open AI Gym toolkit. We’ll then create a Q table of this game using simple Python, and then create a Q network using Keras. If you’d like to scrub up on Keras, check out my introductory Keras tutorial. All code present in this tutorial is available on this site’s Github page.


Recommended online course  If you’re more of a video based learner, I’d recommend the following inexpensive Udemy online course in reinforcement learning: Artificial Intelligence: Reinforcement Learning in Python


Reinforcement learning – the basics

Reinforcement learning can be considered the third genre of the machine learning triad – unsupervised learning, supervised learning and reinforcement learning. In supervised learning, we supply the machine learning system with curated (x, y) training pairs, where the intention is for the network to learn to map x to y. In reinforcement learning, we create an agent which performs actions in an environment and the agent receives various rewards depending on what state it is in when it performs the action. In other words, an agent explores a kind of game, and it is trained by trying to maximize rewards in this game. This cycle is illustrated in the figure below:

Reinforcement learning with Python and Keras - Reinforcement learning environment

Reinforcement learning environment

As can be observed above, the agent performs some action in the environment. An interpreter views this action in the environment, and feeds back an updated state that the agent now resides in, and also the reward for taking this action. The environment is not known by the agent beforehand, but rather it is discovered by the agent taking incremental steps in time. So, for instance, at time the agent, in state $s_{t}$,  may take action a. This results in a new state $s_{t+1}$ and a reward r. This reward can be a positive real number, zero, or a negative real number. It is the goal of the agent to learn which state dependent action to take which maximizes its rewards. The way which the agent optimally learns is the subject of reinforcement learning theory and methodologies.

To more meaningfully examine the theory and possible approaches behind reinforcement learning, it is useful to have a simple example in which to work through. This simple example will come from an environment available on Open AI Gym called NChain.

Open AI Gym example

The NChain example on Open AI Gym is a simple 5 state environment. There are two possible actions in each state, move forward (action 0) and move backwards (action 1). When action 1 is taken, i.e. move backwards, there is an immediate reward of 2 given to the agent – and the agent is returned to state 0 (back to the beginning of the chain). However, when a move forward action is taken (action 0), there is no immediate reward until state 4. When the agent moves forward while in state 4, a reward of 10 is received by the agent. The agent stays in state 4 at this point also, so the reward can be repeated. There is also a random chance that the agent’s action is “flipped” by the environment (i.e. an action 0 is flipped to an action 1 and vice versa). The diagram below demonstrates this environment:

Reinforcement learning - Python and Keras - NChain environment

Open AI Gym’s NChain environment

You can play around with this environment by first installing the Open AI Gym Python package – see instructions here. Then simply open up your Python command prompt and have a play – see the figure below for an example of some of the commands available:

Reinforcement learning - Python and Keras - NChain Python playaround

NChain Python playaround

If you examine the code above, you can observe that first the Python module is imported, and then the environment is loaded via the gym.make() command. The first step is to initalize / reset the environment by running env.reset() – this command returns the initial state of the environment – in this case 0. The first command I then run is env.step(1) – the value in the bracket is the action ID. As explained previously, action 1 represents a step back to the beginning of the chain (state 0). The step() command returns 4 variables in a tuple, these are (in order):

  • The new state after the action
  • The reward due to the action
  • Whether the game is “done” or not – the NChain game is done after 1,000 steps
  • Debugging information – not relevant in this example

As can be observed, starting in state 0 and taking step(1) action, the agent stays in state 0 and gets 2 for its reward. Next, I sent a series of action 0 commands. After every action 0 command, we would expect the progression of the agent along the chain, with the state increasing in increments (i.e. 0 -> 1 -> 2 etc.). However, you’ll observe after the first step(0) command, that the agent stays in state 0 and gets a 2 reward. This is because of the random tendency of the environment to “flip” the action occasionally, so the agent actually performed a 1 action. This is just unlucky.

Nevertheless, I persevere and it can be observed that the state increments as expected, but there is no immediate reward for doing so for the agent until it reaches state 4. When in state 4, an action of 0 will keep the agent in step 4 and give the agent a 10 reward. Not only that, the environment allows this to be done repeatedly, as long as it doesn’t produce an unlucky “flip”, which would send the agent back to state 0 – the beginning of the chain.

Now that we understand the environment that will be used in this tutorial, it is time to consider what method can be used to train the agent.

A first naive heuristic for reinforcement learning

In order to train the agent effectively, we need to find a good policy $\pi$ which maps states to actions in an optimal way to maximize reward. There are various ways of going about finding a good or optimal policy, but first, let’s consider a naive approach.

Let’s conceptualize a table, and call it a reward table, which looks like this:

$$
\begin{bmatrix}
r_{s_0,a_0} & r_{s_0,a_1} \\
r_{s_1,a_0} & r_{s_1,a_1} \\
r_{s_2,a_0} & r_{s_2,a_1} \\
r_{s_3,a_0} & r_{s_3,a_1} \\
r_{s_4,a_0} & r_{s_4,a_1} \\
\end{bmatrix}
$$

Each of the rows corresponds to the 5 available states in the NChain environment, and each column corresponds to the 2 available actions in each state – forward and backward, 0 and 1. The value in each of these table cells corresponds to some measure of reward that the agent has “learnt” occurs when they are in that state and perform that action. So, the value $r_{s_0,a_0}$ would be, say, the sum of the rewards that the agent has received when in the past they have been in state 0 and taken action 0. This table would then let the agent choose between actions based on the summated (or average, median etc. – take your pick) amount of reward the agent has received in the past when taking actions 0 or 1.

This might be a good policy – choose the action resulting in the greatest previous summated reward. Let’s give it a try, the code looks like:

def naive_sum_reward_agent(env, num_episodes=500):
    # this is the table that will hold our summated rewards for
    # each action in each state
    r_table = np.zeros((5, 2))
    for g in range(num_episodes):
        s = env.reset()
        done = False
        while not done:
            if np.sum(r_table[s, :]) == 0:
                # make a random selection of actions
                a = np.random.randint(0, 2)
            else:
                # select the action with highest cummulative reward
                a = np.argmax(r_table[s, :])
            new_s, r, done, _ = env.step(a)
            r_table[s, a] += r
            s = new_s
    return r_table

In the function definition, the environment is passed as the first argument, then the number of episodes (or number of games) that we will train the r_table on. We first create the r_table matrix which I presented previously and which will hold our summated rewards for each state and action. Then there is an outer loop which cycles through the number of episodes. The env.reset() command starts the game afresh each time a new episode is commenced. It also returns the starting state of the game, which is stored in the variable s.

The second, inner loop continues until a “done” signal is returned after an action is passed to the environment. The if statement on the first line of the inner loop checks to see if there are any existing values in the r_table for the current state – it does this by confirming if the sum across the row is equal to 0. If it is zero, then an action is chosen at random – there is no better information available at this stage to judge which action to take.

This condition will only last for a short period of time. After this point, there will be a value stored in at least one of the actions for each state, and the action will be chosen based on which column value is the largest for the row state s. In the code, this choice of the maximum column is executed by the numpy argmax function – this function returns the index of the vector / matrix with the highest value. For example, if the agent is in state 0 and we have the r_table with values [100, 1000] for the first row, action 1 will be selected as the index with the highest value is column 1.

After the action has been selected and stored in a, this action is fed into the environment with env.step(a). This command returns the new state, the reward for this action, whether the game is “done” at this stage and the debugging information that we are not interested in. In the next line, the r_table cell corresponding to state and action is updated by adding the reward to whatever is already existing in the table cell.

Finally the state is updated to new_s – the new state of the agent.

If we run this function, the r_table will look something like:

Examining the results above, you can observe that the most common state for the agent to be in is the first state, seeing as any action 1 will bring the agent back to this point. The least occupied state is state 4, as it is difficult for the agent to progress from state 0 to 4 without the action being “flipped” and the agent being sent back to state 0. You can get different results if you run the function multiple times, and this is because of the stochastic nature of both the environment and the algorithm.

Clearly – something is wrong with this table. One would expect that in state 4, the most rewarding action for the agent would be to choose action 0, which would reward the agent with 10 points, instead of the usual 2 points for an action of 1. Not only that, but it has chosen action 0 for all states – this goes against intuition – surely it would be best to sometimes shoot for state 4 by choosing multiple action 0’s in a row, and that way reap the reward of multiple possible 10 scores.

In fact, there are a number of issues with this way of doing reinforcement learning:

  • First, once there is a reward stored in one of the columns, the agent will always choose that action from that point on. This will lead to the table being “locked in” with respect to actions after just a few steps in the game.
  • Second, because no reward is obtained for most of the states when action 0 is picked, this model for training the agent has no way to encourage acting on delayed reward signal when it is appropriate for it to do so.

Let’s see how these problems could be fixed.

Delayed reward reinforcement learning

If you want to be a medical doctor, you’re going to have to go through some pain to get there. You’ll be studying a long time before you’re free to practice on your own, and the rewards will be low while you are doing so. However, once you get to be a fully fledged MD, the rewards will be great. During your time studying, you would be operating under a delayed reward or delayed gratification paradigm in order to reach that greater reward. However, you might only be willing to undertake that period of delayed reward for a given period of time – you wouldn’t want to be studying forever, or at least, for decades.

We can bring these concepts into our understanding of reinforcement learning. Let’s say we are in state 3 – in the previous case, when the agent chose action 0 to get to state 3, the reward was zero and therefore r_table[3, 0] = 0. Obviously the agent would not see this as an attractive step compared to the alternative for this state i.e. r_table[3, 1] >= 2. But what if we assigned to this state the reward the agent would received if it chose action 0 in state 4? It would look like this: r_table[3, 0] = r + 10 = 10 – a much more attractive alternative!

This idea of propagating possible reward from the best possible actions in future states is a core component of what is called Q learning. In Q learning, the Q value for each action in each state is updated when the relevant information is made available. The Q learning rule is:

$$Q(s, a) = Q(s, a) + \alpha (r + \gamma \max\limits_{a’} Q(s’, a’) – Q(s, a))$$

First, as you can observe, this is an updating rule – the existing Q value is added to, not replaced. Ignoring the $\alpha$ for the moment, we can concentrate on what’s inside the brackets. The first term, ris the reward that was obtained when action a was taken in state s. Next, we have an expression which is a bit more complicated. Ignore the $\gamma$ for the moment and focus on $\max\limits_{a’} Q(s’, a’)$. What this means is that we look at the next state s’ after action a and return the maximum possible Q value in the next state. In other words, return the maximum Q value for the best possible action in the next state. In this way, the agent is looking forward to determine the best possible future rewards before making the next step a.

The $\gamma$ value is called the discounting factor – this decreases the impact of future rewards on the immediate decision making in state s. This is important, as this represents a limited patience in the agent – it won’t study forever to get that medical degree. So $\gamma$ will always be less than 1. The – Q(s, a) term acts to restrict the growth of the Q value as the training of the agent progresses through many iterations. Finally, this whole sum is multiplied by a learning rate $\alpha$ which restricts the updating to ensure it doesn’t “race” to a solution – this is important for optimal convergence (see my  neural networks tutorial for more on learning rate).

Note that while the learning rule only examines the best action in the following state, in reality, discounted rewards still cascade down from future states. For instance, if we think of the cascading rewards from all the 0 actions (i.e. moving forward along the chain) and start at state 3, the Q reward will be $r + \gamma \max_a Q(s’, a’) = 0 + 0.95 * 10 = 9.5$ (with a $\gamma$ = 0.95). If we work back from state 3 to state 2 it will be 0 + 0.95 * 9.5 = 9.025. Likewise, the cascaded, discounted reward from to state 1 will be 0 + 0.95 * 9.025 = 8.57, and so on. Therefore, while the immediate updating calculation only looks at the maximum Q value for the next state, “upstream” rewards that have previously been discovered by the agent still cascade down into the present state and action decision. This is a simplification, due to the learning rate and random events in the environment, but represents the general idea.

Now that you (hopefully) understand Q learning, let’s see what it looks like in practice:

def q_learning_with_table(env, num_episodes=500):
    q_table = np.zeros((5, 2))
    y = 0.95
    lr = 0.8
    for i in range(num_episodes):
        s = env.reset()
        done = False
        while not done:
            if np.sum(q_table[s,:]) == 0:
                # make a random selection of actions
                a = np.random.randint(0, 2)
            else:
                # select the action with largest q value in state s
                a = np.argmax(q_table[s, :])
            new_s, r, done, _ = env.step(a)
            q_table[s, a] += r + lr*(y*np.max(q_table[new_s, :]) - q_table[s, a])
            s = new_s
    return q_table

This function is almost exactly the same as the previous naive r_table function that was discussed. The additions and changes are:

  • The variables y which specifies the discounting factor $\gamma$ and lr which is the Q table updating learning rate
  • The line:
    q_table[s, a] += r + lr*(y*np.max(q_table[new_s, :]) - q_table[s, a])

This line executes the Q learning rule that was presented previously. The np.max(q_table[new_s, :]) is an easy way of selecting the maximum value in the q_table for the row new_s. After this function is run, an example q_table output is:

This output is strange, isn’t it? Again, we would expect at least the state 4 – action 0 combination to have the highest Q score, but it doesn’t.  We might also expect the reward from this action in this state to have cascaded down through the states 0 to 3. Something has clearly gone wrong – and the answer is that there isn’t enough exploration going on within the agent training method.

Q learning with $\epsilon$-greedy action selection

If we think about the previous iteration of the agent training model using Q learning, the action selection policy is based solely on the maximum Q value in any given state. It is conceivable that, given the random nature of the environment, that the agent initially makes “bad” decisions. The Q values arising from these decisions may easily be “locked in” – and from that time forward, bad decisions may continue to be made by the agent because it can only ever select the maximum Q value in any given state, even if these values are not necessarily optimal. This action selection policy is called a greedy policy.

So we need a way for the agent to eventually always choose the “best” set of actions in the environment, yet at the same time allowing the agent to not get “locked in” and giving it some space to explore alternatives. What is required is the $\epsilon$-greedy policy.

The $\epsilon$-greedy policy in reinforcement learning is basically the same as the greedy policy, except that there is a value $\epsilon$ (which may be set to decay over time) where, if a random number is selected which is less than this value, an action is chosen completely at random. This step allows some random exploration of the value of various actions in various states, and can be scaled back over time to allow the algorithm to concentrate more on exploiting the best strategies that it has found. This mechanism can be expressed in code as:

def eps_greedy_q_learning_with_table(env, num_episodes=500):
    q_table = np.zeros((5, 2))
    y = 0.95
    eps = 0.5
    lr = 0.8
    decay_factor = 0.999
    for i in range(num_episodes):
        s = env.reset()
        eps *= decay_factor
        done = False
        while not done:
            # select the action with highest cummulative reward
            if np.random.random() < eps or np.sum(q_table[s, :]) == 0:
                a = np.random.randint(0, 2)
            else:
                a = np.argmax(q_table[s, :])
            # pdb.set_trace()
            new_s, r, done, _ = env.step(a)
            q_table[s, a] += r + lr * (y * np.max(q_table[new_s, :]) - q_table[s, a])
            s = new_s
    return q_table

This code shows the introduction of the $\epsilon$ value – eps. There is also an associated eps decay_factor which exponentially decays eps with each episode eps *= decay_factor. The $\epsilon$-greedy based action selection can be found in this code:

            if np.random.random() < eps or np.sum(q_table[s, :]) == 0:
                a = np.random.randint(0, 2)
            else:
                a = np.argmax(q_table[s, :])

The first component of the if statement shows a random number being selected, between 0 and 1, and determining if this is below eps. If so, the action will be selected randomly from the two possible actions in each state. The second part of the if statement is a random selection if there are no values stored in the q_table so far. If neither of these conditions hold true, the action is selected as per normal by taking the action with the highest q value.

The rest of the code is the same as the standard greedy implementation with Q learning discussed previously. This code produces a q_table which looks something like the following:

Finally we have a table which favors action 0 in state 4 – in other words what we would expect to happen given the reward of 10 that is up for grabs via that action in that state. Notice also that, as opposed to the previous tables from the other methods, that there are no actions with a 0 Q value – this is because the full action space has been explored via the randomness introduced by the $\epsilon$-greedy policy.

Comparing the methods

Let’s see if the last agent training model actually produces an agent that gathers the most rewards in any given game. The code below shows the three models trained and then tested over 100 iterations to see which agent performs the best over a test game. The models are trained as well as tested in each iteration because there is significant variability in the environment which messes around with the efficacy of the training – so this is an attempt to understand average performance of the different models. The main testing code looks like:

def test_methods(env, num_iterations=100):
    winner = np.zeros((3,))
    for g in range(num_iterations):
        m0_table = naive_sum_reward_agent(env, 500)
        m1_table = q_learning_with_table(env, 500)
        m2_table = eps_greedy_q_learning_with_table(env, 500)
        m0 = run_game(m0_table, env)
        m1 = run_game(m1_table, env)
        m2 = run_game(m2_table, env)
        w = np.argmax(np.array([m0, m1, m2]))
        winner[w] += 1
        print("Game {} of {}".format(g + 1, num_iterations))
    return winner

First, this method creates a numpy zeros array of length 3 to hold the results of the winner in each iteration – the winning method is the method that returns the highest rewards after training and playing. The run_game function looks like:

def run_game(table, env):
    s = env.reset()
    tot_reward = 0
    done = False
    while not done:
        a = np.argmax(table[s, :])
        s, r, done, _ = env.step(a)
        tot_reward += r
    return tot_reward

Here, it can be observed that the trained table given to the function is used for action selection, and the total reward accumulated during the game is returned. A sample outcome from this experiment (i.e. the vector w) is shown below:

[13, 22, 65]

 

As can be observed, of the 100 experiments the $\epsilon$-greedy, Q learning algorithm (i.e. the third model that was presented) wins 65 of them. This is followed by the standard greedy implementation of Q learning, which won 22 of the experiments. Finally the naive accumulated rewards method only won 13 experiments. So as can be seen, the $\epsilon$-greedy Q learning method is quite an effective way of executing reinforcement learning.

So far, we have been dealing with explicit tables to hold information about the best actions and which actions to choose in any given state. However, while this is perfectly reasonable for a small environment like NChain, the table gets far too large and unwieldy for more complicated environments which have a huge number of states and potential actions.

This is where neural networks can be used in reinforcement learning. Instead of having explicit tables, instead we can train a neural network to predict Q values for each action in a given state. This will be demonstrated using Keras in the next section.

Reinforcement learning with Keras

To develop a neural network which can perform Q learning, the input needs to be the current state (plus potentially some other information about the environment) and it needs to output the relevant Q values for each action in that state. The Q values which are output should approach, as training progresses, the values produced in the Q learning updating rule. Therefore, the loss or cost function for the neural network should be:

$$\text{loss} = (\underbrace{r + \gamma \max_{a’} Q'(s’, a’)}_{\text{target}} – \underbrace{Q(s, a)}_{\text{prediction}})^2$$

The reinforcement learning architecture that we are going to build in Keras is shown below:

Reinforcement learning Python Keras - architecture

Reinforcement learning Keras architecture

The input to the network is the one-hot encoded state vector. For instance, the vector which corresponds to state 1 is [0, 1, 0, 0, 0] and state 3 is [0, 0, 0, 1, 0]. In this case, a hidden layer of 10 nodes with sigmoid activation will be used. The output layer is a linear activated set of two nodes, corresponding to the two Q values assigned to each state to represent the two possible actions. Linear activation means that the output depends only on the linear summation of the inputs and the weights, with no additional function applied to that summation. For more on neural networks, check out my comprehensive neural network tutorial.

Building this network is easy in Keras – to learn more about how to use Keras, check out my tutorial. The code below shows how it can be done in a few lines:

model = Sequential()
model.add(InputLayer(batch_input_shape=(1, 5)))
model.add(Dense(10, activation='sigmoid'))
model.add(Dense(2, activation='linear'))
model.compile(loss='mse', optimizer='adam', metrics=['mae'])

First, the model is created using the Keras Sequential API. Then an input layer is added which takes inputs corresponding to the one-hot encoded state vectors. Then the sigmoid activated hidden layer with 10 nodes is added, followed by the linear activated output layer which will yield the Q values for each action. Finally the model is compiled using a mean-squared error loss function (to correspond with the loss function defined previously) with the Adam optimizer being used in its default Keras state.

To use this model in the training environment, the following code is run which is similar to the previous $\epsilon$-greedy Q learning methodology with an explicit Q table:

    # now execute the q learning
    y = 0.95
    eps = 0.5
    decay_factor = 0.999
    r_avg_list = []
    for i in range(num_episodes):
        s = env.reset()
        eps *= decay_factor
        if i % 100 == 0:
            print("Episode {} of {}".format(i + 1, num_episodes))
        done = False
        r_sum = 0
        while not done:
            if np.random.random() < eps:
                a = np.random.randint(0, 2)
            else:
                a = np.argmax(model.predict(np.identity(5)[s:s + 1]))
            new_s, r, done, _ = env.step(a)
            target = r + y * np.max(model.predict(np.identity(5)[new_s:new_s + 1]))
            target_vec = model.predict(np.identity(5)[s:s + 1])[0]
            target_vec[a] = target
            model.fit(np.identity(5)[s:s + 1], target_vec.reshape(-1, 2), epochs=1, verbose=0)
            s = new_s
            r_sum += r
        r_avg_list.append(r_sum / 1000)

The first major difference in the Keras implementation is the following code:

            if np.random.random() < eps:
                a = np.random.randint(0, 2)
            else:
                a = np.argmax(model.predict(np.identity(5)[s:s + 1]))

The first condition in the if statement is the implementation of the $\epsilon$-greedy action selection policy that has been discussed already. The second condition uses the Keras model to produce the two Q values – one for each possible state. It does this by calling the model.predict() function. Here the numpy identity function is used, with vector slicing, to produce the one-hot encoding of the current state s. The standard numpy argmax function is used to select the action with the highest Q value returned from the Keras model prediction.

The second major difference is the following four lines:

            target = r + y * np.max(model.predict(np.identity(5)[new_s:new_s + 1]))
            target_vec = model.predict(np.identity(5)[s:s + 1])[0]
            target_vec[a] = target
            model.fit(np.identity(5)[s:s + 1], target_vec.reshape(-1, 2), epochs=1, verbose=0)

The first line sets the target as the Q learning updating rule that has been previously presented. It is the reward r plus the discounted maximum of the predicted Q values for the new state, new_s. This is the value that we want the Keras model to learn to predict for state and action a i.e. Q(s,a). However, our Keras model has an output for each of the two actions – we don’t want to alter the value for the other action, only the action which has been chosen. So on the next line, target_vec is created which extracts both predicted Q values for state s. On the following line, only the Q value corresponding to the action a is changed to target – the other action’s Q value is left untouched.

The final line is where the Keras model is updated in a single training step. The first argument is the current state – i.e. the one-hot encoded input to the model. The second is our target vector which is reshaped to make it have the required dimensions of (1, 2). The third argument tells the fit function that we only want to train for a single iteration and finally the verbose flag simply tells Keras not to print out the training progress.

Running this training over 1000 game episodes reveals the following average reward for each step in the game:

Reinforcement learning Python Keras - training improvement in reward

Reinforcement learning in Keras – average reward improvement over number of episodes trained

As can be observed, the average reward per step in the game increases over each game episode, showing that the Keras model is learning well (if a little slowly).

We can also run the following code to get an output of the Q values for each of the states – this is basically getting the Keras model to reproduce our explicit Q table that was generated in previous methods:

State 0 – action [[62.734287 61.350456]]

State 1 – action [[66.317955 62.27209 ]]

State 2 – action [[70.82501 63.262383]]

State 3 – action [[76.63797 64.75874]]

State 4 – action [[84.51073 66.499725]]

This output looks sensible – we can see that the Q values for each state will favor choosing action 0 (moving forward) to shoot for those big, repeated rewards in state 4. Intuitively, this seems like the best strategy.

So there you have it – you should now be able to understand some basic concepts in reinforcement learning, and understand how to build Q learning models in Keras. This is just scraping the surface of reinforcement learning, so stay tuned for future posts on this topic (or check out the recommended course below) where more interesting games are played!


Recommended online course  If you’re more of a video based learner, I’d recommend the following inexpensive Udemy online course in reinforcement learning: Artificial Intelligence: Reinforcement Learning in Python


 

1,190 thoughts on “Reinforcement learning tutorial using Python and Keras”

  1. Thanks Andy for this comprehensive RL tutorial. The parts read from “Reinforcement Learning: An Introduction” from Sutton and Barto got some substance now 🙂

  2. Thanks fortune. Your article worth a lot more than ALL of lessons I have paid (or freely attended on-line) combined together. Now I can move on strongly with advanced ones. Thank you and please keep writing such great articles.

    Regards,
    Steve

  3. Thank you for this tutorial. It is a great introduction for RL.

    I really enjoyed the progression. First you showed the importance of exploration and then delved into incorporating Keras.

    Good job and keep up the good work!

  4. Andy, really impressive tutorial…
    I’m taking the course on Udemy as cited on your recomendation. It was great too but your article is fantastic in giving the high (and middle) level concepts necessary to understand RL. It is simply an obrigatory read to take off on this subject.
    Really thanks

  5. Yeah I have to chip in, great tutorial! You know, not everyone has the gift to explain principles and basics in a way so that the concept is conveyed and the student gets slightly booted and prepared for more advanced stuff! Cudos to you! +++

  6. I’ve seen multiple tutorials on the topic and by far this was the one which explained it in the most understandable way, by showing the steps and where the NN go into the topic. Thank you for your work

  7. I like what you guys are up also. Such clever work and reporting! Keep up the superb works guys I have incorporated you guys to my blogroll. I think it will improve the value of my site 🙂

  8. I am extremely impressed together with your writing
    abilities as smartly as with the format
    to your weblog. Is this a paid theme or did you modify it yourself?
    Anyway keep up the nice high quality writing, it’s rare to peer a nice blog like this one
    nowadays..

  9. The other day, while I was at work, my cousin stole
    my iPad and tested to see if it can survive a 25 foot drop, just so she can be
    a youtube sensation. My apple ipad is now destroyed and she has 83 views.
    I know this is totally off topic but I had to share it with someone!

    Also visit my page – fast weight loss

  10. Wow, amazing weblog format! How lengthy have you ever been blogging for?
    you made blogging glance easy. The full glance of your website is fantastic,
    let alone the content![X-N-E-W-L-I-N-S-P-I-N-X]I simply could not depart your
    website prior to suggesting that I actually enjoyed the usual info an individual supply on your guests?

    Is gonna be back continuously in order to inspect new posts.

    my webpage … Nouvee Skin Cream

  11. Thank you, I have recently been looking for information approximately this topic for ages and yours
    is the greatest I’ve came upon so far. However, what concerning the conclusion? Are you
    positive concerning the source?

    Here is my blog post – Pinnacle Science Testo Boost (penzu.com)

  12. of course like your web site but you need to check the spelling
    on quite a few of your posts. Many of them are rife with spelling issues and I in finding it very troublesome to inform
    the truth on the other hand I’ll certainly come back again.

    Here is my page … kids smoking

  13. Hi there, just became aware of your blog through Google, and found that
    it is truly informative. I am gonna watch out for brussels.
    I will appreciate if you continue this in future. Many people will be benefited from your writing.
    Cheers!

    Feel free to visit my web page :: Nouvee Skin Cream; Kaitlyn,

  14. Thanks for your personal marvelous posting!

    I genuinely enjoyed reading it, you happen to be a great author.I will make sure
    to bookmark your blog and will come back someday. I want to encourage you to continue your
    great job, have a nice day!

    Also visit my web blog :: seed bank

  15. Awesome website you have here but I was wondering if you knew of any
    forums that cover the same topics talked about in this article?
    I’d really love to be a part of community where I can get opinions
    from other knowledgeable people that share the same interest.
    If you have any recommendations, please let me know. Thanks!

    my blog :: seeds prior

  16. Good – I should certainly pronounce, impressed with your web site.
    I had no trouble navigating through all the tabs as well as related information ended up being truly easy
    to do to access. I recently found what I hoped for before you
    know it in the least. Quite unusual. Is likely to appreciate it for
    those who add forums or anything, site theme . a tones way for your client to communicate.
    Excellent task.

    my web-site … http://www.aniene.net

  17. I have been exploring for a bit for any high-quality articles or blog posts on this kind of house .
    Exploring in Yahoo I ultimately stumbled upon this web site.
    Reading this info So i am glad to exhibit that I have
    a very excellent uncanny feeling I found out just what I needed.
    I most unquestionably will make certain to do not put out of your mind
    this website and provides it a look regularly.

    Review my webpage … Pinnacle Science Testo Boost, Leatha,

  18. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.

    I’ve been looking for a plug-in like this for quite some time and was
    hoping maybe you would have some experience with something like this.
    Please let me know if you run into anything. I truly
    enjoy reading your blog and I look forward to your new updates.

    Here is my web page :: Circadiyin (groups.google.com)

  19. I do agree with all of the concepts you’ve offered in your post.
    They are really convincing and will definitely work. Still, the posts are too short for beginners.

    May you please prolong them a bit from next time?
    Thank you for the post.

    Also visit my homepage crash diets

  20. Hi there, I discovered your web site by means of Google while searching for a
    related matter, your website got here up, it appears to be like good.
    I have bookmarked it in my google bookmarks.[X-N-E-W-L-I-N-S-P-I-N-X]Hi there, just changed into alert to your weblog via Google, and located that it is really informative.

    I’m going to be careful pill for adhd brussels.

    I will be grateful in the event you continue this in future.
    A lot of folks might be benefited out of your writing. Cheers!

  21. This is very interesting, You are an overly skilled blogger.
    I’ve joined your rss feed and look forward to looking for more of your magnificent post.
    Also, I’ve shared your web site in my social networks!

    Feel free to surf to my webpage; Pinnacle Science Testo Boost (Morgan)

  22. With havin so much content and articles do you ever run into any issues of plagorism or copyright violation? My site has a lot of completely unique content I’ve
    either authored myself or outsourced but it appears a lot of it is popping it up all over the web without my permission. Do you know any solutions to help prevent content from being ripped
    off? I’d truly appreciate it.

    Look at my web site … https://groups.google.com/g/cannabidoil-reviews/c/tygx9Ych9sQ

  23. What i don’t understood is actually how you’re now not actually much more neatly-appreciated than you might
    be right now. You are very intelligent. You know
    therefore significantly in relation to this subject, produced me in my opinion imagine it from numerous numerous angles.
    Its like women and men are not interested until
    it is something to accomplish with Lady gaga! Your individual stuffs nice.
    Always deal with it up!

    Here is my site … Circadiyin (Bernd)

  24. When someone writes an piece of writing he/she maintains the image of a user in his/her mind that
    how a user can be aware of it. Therefore that’s why this paragraph is great.
    Thanks!

    Feel free to surf to my web-site Circadiyin

  25. What i don’t realize is in fact how you’re now not actually much
    more well-favored than you may be now. You are
    very intelligent. You realize therefore significantly relating to this subject, produced me in my opinion imagine it from a lot of numerous
    angles. Its like women and men aren’t interested unless it’s something
    to do with Lady gaga! Your individual stuffs great.

    Always deal with it up!

    Also visit my blog post :: Circadiyin (yclas380.00web.net)

  26. Hmm it appears like your website ate my first comment (it was super long) so I guess I’ll just sum it up what I wrote and
    say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to everything.
    Do you have any recommendations for novice blog
    writers? I’d definitely appreciate it.

    My blog :: seed sprouts

  27. After looking into a handful of the blog posts on your web page, I truly
    like your technique of writing a blog. I book-marked it to my bookmark website list and will
    be checking back in the near future. Please check out my website
    as well and tell me your opinion.

    Visit my webpage … Circadiyin

  28. Thanks for every one of your labor on this site.
    Betty really loves participating in research and it’s obvious why.
    Most people hear all about the dynamic manner you render
    practical ideas by means of your web site and as well cause participation from other individuals
    on that topic while our favorite daughter is always being taught a lot of things.
    Enjoy the remaining portion of the new year.
    You have been doing a very good job.

    my site; personal cannabis seeds

  29. My brother recommended I may like this blog. He was once totally right.
    This publish actually made my day. You cann’t consider simply how much time I had spent for this information! Thank you!

    Feel free to surf to my homepage :: tips honey

  30. Hi there, just became aware of your blog through Google, and found that it is really informative.
    I?m gonna watch out for brussels. I?ll be grateful if you continue this in future.
    Lots of people will be benefited from your writing.
    Cheers!

    Feel free to visit my site … Nouvee Skin Cream

  31. An outstanding share! I’ve just forwarded this onto a friend who was conducting a little homework on this.
    And he in fact ordered me lunch because I stumbled upon it for
    him… lol. So let me reword this…. Thank YOU for the meal!!
    But yeah, thanx for spending some time to talk about this subject here on your
    website.

    Review my web-site: sleep weight loss

  32. Hey there! This is my first comment here so I just wanted to give
    a quick shout out and say I really enjoy reading through your posts.
    Can you suggest any other blogs/websites/forums that
    go over the same subjects? Many thanks!

    Check out my web site; Pinnacle Science Testo Boost (justpaste.it)

  33. I know this if off topic but I’m looking into starting my own blog
    and was wondering what all is required to get set up?
    I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very internet smart so I’m not 100% sure.
    Any recommendations or advice would be greatly appreciated.
    Appreciate it

    Also visit my webpage; https://cannabidiol-reviews.blogspot.com/2021/09/keoni-cbd-take-care-of-your-body.html [cannabidiol-reviews.blogspot.com]

  34. I really love your site.. Great colors & theme. Did you make this website yourself?
    Please reply back as I?m hoping to create my own website and would like to know where
    you got this from or what the theme is named.

    Cheers!

    Review my blog: high fat

  35. I think what you posted made a lot of sense. However, think on this, suppose you added a little content?
    I ain’t suggesting your information isn’t good.,
    however suppose you added something to possibly get people’s attention? I mean Reinforcement
    learning tutorial using Python and Keras – Adventures
    in Machine Learning is a little boring. You could
    peek at Yahoo’s home page and note how they
    create post titles to get viewers to open the links. You might try adding a video or a related pic or two to get people
    interested about what you’ve written. In my opinion, it would bring your posts a little bit more interesting.

    Also visit my web page: bbs.yunweishidai.com

  36. With havin so much written content do you ever run into any problems of plagorism or copyright infringement?
    My site has a lot of completely unique content I’ve
    either written myself or outsourced but it seems
    a lot of it is popping it up all over the internet without my authorization.
    Do you know any ways to help reduce content from being ripped off?

    I’d really appreciate it.

    Have a look at my web blog – http://keonicbdgummy.ampblogs.com/Keoni-CBD-Take-Care-Of-Your-Body-Naturally–39473668 [http://keonicbdgummy.ampblogs.com/]

  37. Hi I am so glad I found your weblog, I really found you by mistake, while I was looking on Yahoo for something else, Anyways I am here
    now and would just like to say thank you for a fantastic post and a
    all round interesting blog (I also love the theme/design), I don’t have time to read it all at the moment but I have book-marked it and
    also added your RSS feeds, so when I have time I will be back to
    read more, Please do keep up the superb job.

    Here is my webpage: Diaetoxil

  38. Hello I am so thrilled I found your blog page, I really found you by accident,
    while I was searching on Yahoo for something else, Anyhow I am here now and would just like to say thank you for
    a fantastic post and a all round entertaining blog (I also love the theme/design), I don’t have time to go through it all at the moment but I have saved it
    and also added in your RSS feeds, so when I have time I will be back to
    read a lot more, Please do keep up the awesome work.

    Also visit my web site :: protein diet

  39. Whats up very nice blog!! Man .. Excellent ..

    Wonderful .. I will bookmark your website and take the feeds additionally?I am happy to search out numerous useful information right here within the publish, we’d
    like develop extra strategies on this regard, thanks for sharing.

    Here is my web-site :: robert atkins

  40. Hi there, just became alert to your blog through Google,
    and found that it’s really informative. I?m gonna
    watch out for brussels. I will appreciate if you continue this in future.
    A lot of people will be benefited from your writing.
    Cheers!

    Feel free to surf to my web page Nouvee Skin Cream

  41. Thank you so much with regard to giving me personally
    an update on this subject matter on your web-site. Please realize that if a brand new post becomes available or in case
    any improvements occur on the current article, I would consider reading more and finding
    out how to make good use of those techniques you discuss.
    Thanks for your time and consideration of other folks by making this website
    available.

    Here is my web blog … Nouvee Skin Cream (Stevie)

  42. I am only writing to let you be aware of of the really good experience my girl went through reading through
    your web site. She even learned many pieces,
    with the inclusion of what it’s like to have a great teaching mindset
    to get many more smoothly learn about selected specialized things.
    You undoubtedly surpassed her desires. Many thanks for imparting those necessary, healthy,
    edifying and unique tips about your topic to Janet.

    Feel free to surf to my homepage Pinnacle
    Science Testo Boost (Polly)

  43. Appreciating the hard work you put into your website and detailed information you present.
    It’s great to come across a blog every once in a while that isn’t the same unwanted
    rehashed material. Fantastic read! I’ve saved your
    site and I’m adding your RSS feeds to my Google account.

    Feel free to surf to my web blog :: hypnotronstudios.com

  44. Spot on with this write-up, I honestly believe this amazing site needs a great deal more attention. I’ll probably be
    back again to read through more, thanks for the info!

    Feel free to visit my webpage; Diaetoxil (penzu.com)

  45. Whoah this weblog is wonderful i really like reading your posts.
    Keep up the good paintings! You know, many individuals are hunting round for this info,
    you can help them greatly.

    My blog Circadiyin (https://docs.google.com/presentation/d/e/2PACX-1vSge9NF3UHjPF–E4ojIsOXoy53xZ1oNdxZ5DUAeaWSqfJPVRFuTduNEWbnYF64-QUpBilxnrlFsGKb/pub?start=false&loop=false&delayms=3000)

  46. Hello, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam remarks?
    If so how do you protect against it, any plugin or anything you can suggest?
    I get so much lately it’s driving me crazy so any assistance is very much appreciated.

    my homepage :: Nouvee Skin Cream, Blair,

  47. Thanks a bunch for sharing this with all folks you really know what you
    are speaking approximately! Bookmarked. Kindly additionally talk over with my site =).
    We will have a hyperlink exchange contract among
    us

    Feel free to surf to my web-site – Green Roads CBD (Cora)

  48. Please let me know if you’re looking for a writer for your blog.

    You have some really good articles sleep and exercise I feel I would be
    a good asset. If you ever want to take some of the load off, I’d
    really like to write some material for your blog
    in exchange for a link back to mine. Please send me an email
    if interested. Regards!

  49. Hi there! This blog post could not be written any better!
    Reading through this post reminds me of my previous roommate!
    He always kept preaching about this. I most certainly will send this information to him.
    Pretty sure he will have a good read. Many thanks for sharing!

    Also visit my website … Quick Trim Keto – Laverne

  50. Good day! Do you know if they make any plugins to help with SEO?
    I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good results.
    If you know of any please share. Cheers!

    Review my web page Katrina

  51. I must thank you for the efforts you have put in penning this blog.
    I’m hoping to view the same high-grade blog posts by you later on as well.
    In fact, your creative writing abilities has motivated me to
    get my own website now 😉

    Have a look at my webpage – Lourdes

  52. I’m not sure why but this site is loading extremely slow for
    me. Is anyone else having this issue or is it a issue on my end?

    I’ll check back later on and see if the problem still exists.

  53. You most likely make this seem quite simple with your looks but My significant other and i also find this type
    of matter to become actually an issue that I think I would never value.
    It seems also complex and intensely broad for me personally.
    I am looking forward to your future post, I could try to get
    used to it!

  54. I wanted to follow up and allow you to know how considerably I liked discovering your web site today.
    We would consider it a good honor to operate at my office and be able to make use of the tips
    provided on your blog and also engage in visitors’ feedback like this.

    Should a position involving guest author become offered at your end, you should let me know.

    my web blog … ketogenic diet ulitmate

  55. hello there and thank you for your information ? I have
    definitely picked up anything new from right here. I did however expertise several technical points using this website, as I experienced to
    reload the web site lots of times previous to
    I could get it to load correctly. I had been wondering if your
    web host is OK? Not that I am complaining, but slow loading instances times will often affect
    your placement in google and can damage your high quality score if advertising
    and marketing with Adwords. Well I am adding this
    RSS to my email and can look out for a lot more of your respective fascinating content.
    Ensure that you update this again very soon.

    Take a look at my blog casualvalueinvestor.com

  56. We’re a group of volunteers and starting a new scheme in our community.
    Your website provided us with valuable information to work on. You’ve done an impressive job and our entire community
    will be thankful to you.

    My blog – oil swishing

  57. Hi! Do you know if they make any plugins to assist with Search Engine Optimization? I’m trying to get my blog to rank for some
    targeted keywords but I’m not seeing very good gains.
    If you know of any please share. Thank you!

    Also visit my web blog; carb diet

  58. I seldom comment, but i did some searching and wound up here Reinforcement learning tutorial using
    Python and Keras – Adventures in Machine Learning.
    And I do have a couple of questions for you if you don’t mind.
    Could it be simply me or does it give the impression like a few of the comments come
    across like coming from brain dead people? 😛 And, if you
    are posting on other online sites, I’d like to follow anything new you have
    to post. Would you list of every one of all your community pages like your linkedin profile, Facebook page or
    twitter feed?

    My web site: seo tips

  59. I like the helpful information you provide in your articles.
    I will bookmark your weblog and check again here regularly.

    I’m quite certain I’ll learn a lot of new stuff
    right here! Best of luck for the next!

    my blog … Susanne

  60. Fantastic website you have here but I was wanting to know if you knew of
    any message boards that cover the same topics discussed in this article?
    I’d really love to be a part of online community where I can get feed-back from
    other experienced people that share the same interest. If you have any suggestions,
    please let me know. Thanks a lot!

    Visit my site: lose weight fast

  61. I actually still can not quite assume that I could possibly be one of those
    reading through the important guidelines found on your website.
    My family and I are truly thankful for the generosity and for offering me the advantage to pursue the chosen career path.
    Thanks for the important information I got from your web-site.

    Review my page … Green Roads CBD (Raina)

  62. hey there and thank you for your info ? I have certainly picked up anything new from right here.
    I did however expertise some technical points using this website, since
    I experienced to reload the web site many times previous to I could get it
    to load correctly. I had been wondering if your web host is
    OK? Not that I am complaining, but sluggish loading instances times
    will often affect your placement in google and can damage your high-quality score if advertising and marketing
    with Adwords. Well I am adding this RSS to my e-mail and can look out for a lot more of your respective exciting content.
    Ensure that you update this again soon.

    Also visit my blog flaxseed oil

  63. I would like to thank you for the efforts you have put in penning this blog.
    I am hoping to check out the same high-grade content from you in the future as well.
    In truth, your creative writing abilities has motivated me to get my
    very own website now 😉

    Feel free to surf to my website: bbs.inhe365.com

  64. We still cannot quite assume that I could end up being one of
    those reading the important guidelines found on your web blog.
    My family and I are truly thankful on your generosity and for presenting
    me the chance to pursue my personal chosen profession path.
    Thank you for the important information I got from your blog.

    Feel free to surf to my website: Green Roads CBD

  65. Thank you a bunch for sharing this with all people you actually recognize what you’re talking about!
    Bookmarked. Please additionally visit my site =). We can have a hyperlink alternate
    arrangement between us

    Also visit my homepage Dessie

  66. I’m really enjoying the design and layout of your blog.
    It’s a very easy on the eyes which makes it much more pleasant for me to
    come here and visit more often. Did you hire out a designer to create your theme?
    Superb work!

    my webpage lower ldl

  67. Thank you for the sensible critique. Me and my neighbor were just preparing to do
    a little research about this. We got a grab a book from
    our local library but I think I learned more clear from this post.
    I am very glad to see such excellent information being shared freely out
    there.

    Also visit my web page … http://www.aniene.net

  68. Howdy! I know this is sort of off-topic however I had to ask.
    Does running a well-established website such as yours take a massive amount work?
    I’m brand new to blogging but I do write in my diary daily.
    I’d like to start a blog so I can share my personal experience
    and views online. Please let me know if
    you have any recommendations or tips for brand new aspiring blog owners.
    Appreciate it!

    Here is my web page compare diet pills

  69. I just wanted to thank you yet again for this amazing
    web-site you have developed here. It can be full of ideas for those who are really interested in this subject, especially
    this very post. Your all so sweet and thoughtful of others as
    well as reading the blog posts is a good delight in my opinion. And exactly what a generous
    gift! Ben and I are going to have excitement making use of your guidelines in what we have to do in the near
    future. Our listing is a kilometer long so your tips
    might be put to good use.

    My blog post :: skin care p

  70. Wonderful goods from you, man. I have understand your stuff previous to
    and you are just extremely wonderful. I actually like what
    you have acquired here, certainly like what you’re saying and the way in which you say it.
    You make it enjoyable and you still take care of to keep it smart.
    I can not wait how to stop smoking weed
    read much more from you. This is really a tremendous site.

  71. I loved as much as you will receive carried out right here.
    The sketch is tasteful, your authored subject matter stylish.

    nonetheless, you command get bought an edginess over that
    you wish be delivering the following. unwell unquestionably come more formerly again as exactly
    the same nearly very often inside case you shield this hike.

    Also visit my web page … carb cycling

  72. I not to mention my friends happened to be examining the
    excellent helpful hints from the blog and then before long came up
    with an awful feeling I never thanked the web blog owner for those secrets.
    Most of the young boys were definitely as a consequence warmed to learn them and have
    now honestly been making the most of those things.
    Appreciation for genuinely really thoughtful and also
    for figuring out this kind of very good areas millions of individuals are really wanting
    to know about. Our honest regret for not expressing gratitude to you earlier.

    Take a look at my blog :: mashed potato diet

  73. We absolutely love your blog and find most of your post’s to be just what I’m looking for.

    Do you offer guest writers to write content for you?
    I wouldn’t mind writing a post or elaborating on many of the
    subjects you write with regards to here. Again, awesome web log!

    my page … http://www.aniene.net

  74. I think that what you wrote made a lot of sense.
    However, what about this? what if you were to write a killer
    headline? I am not saying your content isn’t solid, however suppose you added something that makes people desire more?
    I mean Reinforcement learning tutorial using Python and Keras –
    Adventures in Machine Learning is a little plain. You ought to glance at Yahoo’s home page and note how they
    create post titles to get people to click. You might add a related video or a picture
    or two to grab people excited about everything’ve written. Just my opinion, it could make your posts a
    little livelier.

    Feel free to surf to my web site: Bobbye

  75. Hello There. I found your blog using msn. This is a very smartly written article.
    I will be sure to bookmark it and come back to read extra of your useful information. Thanks for the post.
    I will definitely comeback.

    Here is my web page growing weed

  76. I was honored to receive a call from a friend as he uncovered the important suggestions shared on your own site.
    Examining your blog publication is a real great experience.
    Thanks again for thinking about readers like me, and I want for
    you the best of achievements as a professional in this field.

    my web blog … high fat

  77. I know this if off topic but I’m looking into starting my own blog and was curious what all is needed to get set up?
    I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very internet savvy so I’m not 100% positive. Any suggestions or advice would be greatly appreciated.
    Many thanks

    Here is my blog: http://www.comptine.biz

  78. Right here is the right blog for anyone who wants to understand this
    topic. You realize a whole lot its almost hard to argue with you
    (not that I actually would want to?HaHa). You definitely put a brand new spin on a topic that’s been discussed for many years.
    Great stuff, just great!

    Here is my web site – cleveland clinic diet

  79. Terrific paintings! This is the kind of info that are supposed to be shared across the
    internet. Disgrace on the search engines for not positioning
    this publish higher! Come on over and consult with my web site .
    Thanks =)

    Here is my homepage; 163.30.42.16

  80. Hey! I know this is kinda off topic but I’d figured I’d ask.
    Would you be interested in trading links or maybe guest authoring a blog article or vice-versa?
    My website addresses a lot of the same topics as yours and I believe we could greatly benefit from each
    other. If you are interested feel free to shoot me
    an email. I look forward to hearing from you! Terrific blog by
    the way!

    Take a look at my site – http://www.fotosombra.com.br

  81. Howdy! This blog post couldn?t be written much better!
    Looking at this article reminds me of my previous roommate!
    He continually kept talking about this. I’ll send this information to him.

    Pretty sure he’s going to have a great read. Thank you for sharing!

    My website: point sleep

  82. I know this if off topic but I’m looking into starting my own blog and was wondering what
    all is needed to get set up? I’m assuming having a blog like
    yours would cost a pretty penny? I’m not very internet smart so I’m not 100% sure.
    Any tips or advice would be greatly appreciated. Thank you

    My web blog hemp seed oil uses

  83. Hello there, There’s no doubt that your website may be having browser compatibility issues.
    Whenever I look at your site in Safari, it looks fine but when opening in Internet
    Explorer, it has some overlapping issues. I simply wanted to give you a quick heads up!

    Besides that, wonderful website!

    Feel free to visit my web-site: eczema on feet

  84. Thank you for the sensible critique. Me and
    my neighbor were just preparing to do a little research about this.
    We got a grab a book from our area library but I think I learned more from this post.
    I’m very glad to see such excellent info being shared freely
    out there.

    Here is my page – diet pill

  85. Everything posted was very logical. But, think about this,
    suppose you typed a catchier title? I ain’t suggesting your content isn’t solid.,
    however what if you added a title that makes people desire more?
    I mean Reinforcement learning tutorial using Python and Keras – Adventures in Machine Learning
    is a little boring. You could peek at Yahoo’s front page
    and watch how they create post titles to grab people to click.
    You might add a video or a related picture or
    two to get people interested about everything’ve got to say.
    In my opinion, it might bring your posts a
    little livelier.

    my webpage; cannabis dispensaries-san

  86. Fantastic goods from you, man. I have have in mind your stuff previous to and you’re just
    too great. I really like what you’ve received here, certainly like what you
    are saying and the way in which in which you assert it.
    You are making it entertaining and you still care for to keep it
    sensible. I cant wait to read much more from you. This is actually a great web site.

    Also visit my web blog; accessing medical cannabis

  87. Its like you learn my mind! You appear to
    understand so much about this, like you wrote the ebook
    in it or something. I feel that you just could do with a few %
    to force the message house a bit, however other than that, that
    is great blog. An excellent read. I’ll definitely be
    back.

    My homepage heal eczema

  88. Fantastic beat ! I would like to apprentice while you amend your web site, how can i subscribe for a blog web site?
    The account aided me a acceptable deal. I had been a little
    bit acquainted of this your broadcast provided bright clear
    concept

    Feel free to visit my website – low carb diet

  89. May I simply say what a comfort to uncover an individual who actually understands what they’re talking about over the internet.
    You actually understand how to bring an issue to light and make it important.

    More people should check this out and understand this side of the
    story. I was surprised you’re not more popular given that you certainly
    possess the gift.

    Also visit my web-site – eczema remedies

  90. I don’t know if it’s just me or if everyone else
    encountering issues with your blog. It appears as
    though some of the written text on your content are running
    off the screen. Can somebody else please provide feedback and let me know if this is happening to them as well?
    This may be a issue with my browser because I’ve had this happen before.
    Thanks

    My homepage: french knife

  91. May I simply say what a comfort to find an individual who
    really knows what they’re discussing on the
    web. You definitely know how to bring an issue to light and
    make it important. More and more people need to read this and understand this side of your
    story. It’s surprising you are not more popular since you most
    certainly have the gift.

    my web site growing weed indoors

  92. Hello there, I discovered your web site by the use of Google at the
    same time as searching for a comparable topic, your web
    site came up, it seems to be great. I’ve bookmarked it in my google bookmarks.[X-N-E-W-L-I-N-S-P-I-N-X]Hello there, simply was
    aware of your weblog via Google, and found that it is really informative.
    I’m gonna be careful for brussels. I will be grateful if you
    continue this in future. Many folks will be benefited out of your writing.
    Cheers!

    my homepage … low carbohydrate

  93. I?m impressed, I have to admit. Seldom do I encounter a blog that?s
    both educative and engaging, and without a doubt, you have
    hit the nail on the head. The problem is something that too few people are speaking intelligently about.
    Now i’m very happy that I found this in my hunt for something regarding this.

    Here is my webpage – forum.canerildes.com

  94. I was just searching for this information for a while. After six
    hours of continuous Googleing, at last I got it in your site.
    I wonder what’s the lack of Google strategy that do not rank this
    type of informative websites in top of the list.
    Generally the top sites are full of garbage.

    Look into my web blog: tips honey

  95. I just like the helpful info you supply for your articles.

    I’ll bookmark your blog and test again here frequently. I am fairly
    sure I’ll be informed lots of new stuff right here!
    Good luck for the following!

    Look into my page Niamh

  96. Its such as you learn my thoughts! You seem to grasp so much approximately this,
    such as you wrote the e-book in it or something.
    I think that you simply can do with some percent to power the message house a little bit,
    but instead of that, that is fantastic blog. A fantastic read.
    I will definitely be back.

  97. What i don’t realize is if truth be told how you’re now not actually much more neatly-liked than you may be now.
    You are very intelligent. You realize thus considerably in terms of this topic, produced me
    personally believe it from numerous numerous
    angles. Its like men and women aren’t fascinated until it is something to
    do with Lady gaga! Your personal stuffs nice. All the time care for it up!

    my website – purchase hemp

  98. I have been surfing online more than 2 hours today, yet I never found any interesting article like yours.
    It is pretty worth enough for me. In my opinion, if all web owners and bloggers made
    good content as you did, the internet will be a lot more useful than ever before.

    Here is my website lose weight safely

  99. I’ve been exploring for a little bit for any high quality articles or weblog posts in this
    sort of space . Exploring in Yahoo I eventually stumbled upon this site.
    Reading this info So i am happy to convey that I’ve an incredibly just right uncanny feeling
    I discovered exactly what I needed. I most without a doubt will make
    sure to do not fail to remember this website and provides it a look regularly.

    Here is my web-site – hemp seed contains

  100. Hi! I know this is kinda off topic however I’d figured I’d ask.
    Would you be interested in trading links or maybe guest authoring a blog post or vice-versa?
    My website discusses a lot of the same topics as yours and I think
    we could greatly benefit from each other. If you
    are interested feel free to shoot me an e-mail.
    I look forward to hearing from you! Fantastic blog by the way!

    Here is my homepage :: stop smoking weed today

  101. It’s perfect time to make some plans for the future and it’s
    time to be happy. I’ve read this post and if I
    could I desire to suggest you few interesting
    things or suggestions. Perhaps you can write next articles referring to this article.
    I want to read even more things about it!

    my web site: weed doctor

  102. I have to point out my appreciation for your kindness giving support to
    individuals who really need help on your theme. Your very own commitment to passing the solution around was exceedingly productive and have usually made many people just like me to attain their endeavors.
    Your personal useful instruction entails much a person like me and further more
    to my colleagues. Warm regards; from all of us.

    Look into my web site: everyday sex

  103. It is perfect time to make some plans for the future and it is time to be happy.
    I have read this post and if I could I desire to suggest
    you few interesting things or tips. Perhaps you
    can write next articles referring to this article.
    I want to read more things about it!

    Here is my page – Emile

  104. Just wish to say your article is as surprising. The clarity in your post
    is just cool and i can assume you’re an expert on this subject.

    Fine with your permission let me to grab your RSS feed to keep updated with forthcoming
    post. Thanks a million and please carry on the enjoyable work.

    Check out my page keto diet

  105. It’s the best time to make some plans for the future
    and it’s time to be happy. I have read this post and if I could I want to suggest you some interesting things
    or suggestions. Perhaps you could write next articles referring to this article.
    I desire to read even more things about it!

    Also visit my web page – term treatment process

  106. I don’t know whether it’s just me or if perhaps everyone else encountering problems with your blog.
    It appears as though some of the text in your content are running off the screen. Can someone else please
    comment and let me know if this is happening to them as well?
    This could be a issue with my web browser because I’ve had
    this happen previously. Appreciate it

    Also visit my web page good knife

  107. Hi there just wanted to give you a quick heads up and let you know a few of the pictures
    aren’t loading correctly. I’m not sure why but I think its a linking issue.
    I’ve tried it in two different browsers and both show the same outcome.

    Have a look at my website – weight watchers (ibbs.uu.cc)

  108. I’m amazed, I must say. Seldom do I come across a blog
    that’s both equally educative and interesting, and without a doubt, you’ve hit the nail on the head.
    The issue is an issue that not enough people are speaking intelligently about.
    Now i’m very happy that I came across this in my search for something relating to this.

    my homepage … good balanced diet [http://www.zgyssyw.com/home.php?mod=space&uid=802610&do=profile]

  109. Thanks a ton for being my own mentor on this topic.
    My spouse and i enjoyed your article very much and most of all enjoyed
    reading how you really handled the aspect I considered to be controversial.

    You happen to be always extremely kind to readers like me
    and help me in my lifestyle. Thank you.

    my blog post :: healthy foods

  110. I absolutely love your blog and find most of your post’s to be exactly what I’m looking for.
    Do you offer guest writers to write content available for you?
    I wouldn’t mind writing a post or elaborating on most of the subjects you write regarding here.
    Again, awesome weblog!

    My web blog … oily skin

  111. I absolutely love your blog and find almost all of your post’s to be exactly what I’m looking
    for. can you offer guest writers to write content in your case?
    I wouldn’t mind composing a post or elaborating on a few of the subjects
    you write in relation to here. Again, awesome site!

    Stop by my web page :: sylvbuster.free.fr

  112. After I originally left a comment I seem to have clicked the -Notify me when new
    comments are added- checkbox and from now on whenever a comment
    is added I receive four emails with the same comment.
    There has to be a means you are able to remove me from
    that service? Thanks a lot!

    Also visit my homepage: night skin

  113. Hey there just wanted to give you a quick heads
    up. The words in your content seem to be running off the screen in Internet
    explorer. I’m not sure if this is a formatting issue or something to do with browser compatibility but I thought I’d post
    to let you know. The design look great though!
    Hope you get the problem resolved soon. Cheers

    My homepage low libido

  114. I was just looking for this info for a while. After six hours of
    continuous Googleing, at last I got it in your site. I wonder what is the lack of
    Google strategy that don’t rank this type of informative
    websites in top of the list. Generally the top web sites are full
    of garbage.

    Here is my homepage :: christmas weight gain

  115. You truly make that seem super easy with your demonstrating but I truly find
    this sort of matter staying actually a thing that I
    think Let me never understand. It seems as well complex and
    in addition broad to me. I am looking forward for your next post, Sick and tired try to get utilized to it!

  116. hello there and thank you for your information ?
    I have definitely picked up anything new from right here.
    I did however expertise a few technical points
    using this web site, since I experienced to reload the website many times previous to I could get it to load correctly.
    I had been wondering if your web host is OK? Not that I’m complaining, but
    slow loading instances times will sometimes affect your placement in google and
    can damage your quality score if advertising and marketing with Adwords.
    Anyway I am adding this RSS to my e-mail and can look out for much more what types of diets are best for our bodies your respective fascinating content.

    Make sure you update this again soon.

  117. Valuable information. Fortunate me I discovered your
    website accidentally, and I am surprised why this accident did not came about in advance!
    I bookmarked it.

    Look at my web-site :: cleveland clinic diet – Winifred,

  118. I would like to show thanks to you just for bailing me out of this particular issue.
    After looking through the the net and obtaining strategies
    that were not powerful, I assumed my life was gone.
    Being alive without the strategies to the difficulties you’ve resolved all through your main posting is a critical case, as well as
    those which might have badly affected my career if
    I hadn’t encountered your web site. That skills and kindness in handling all things was valuable.
    I’m not sure what I would have done if I hadn’t come across such a thing like this.

    It’s possible to at this time look ahead to my future.

    Thanks so much for this specialized and effective guide. I will not hesitate to recommend your web sites to any person who needs counselling about this problem.

    Also visit my web blog … high fat

  119. We wish to thank you once again for the lovely ideas you gave
    Jeremy when preparing her own post-graduate research as well as,
    most importantly, for providing many of the ideas in one blog post.
    If we had known of your web site a year ago, i’d have been kept from the unwanted measures we were employing.
    Thank you very much.

    my web site; find easy diets