Remove stale resque workers using bash

I run a Rails application that uses Resque to manage background jobs.  It works beautifully.  Unfortunately, sometimes resque workers become stuck while working on a task.  This might happen for a number of reasons, but the end result is that you’re missing a worker and you might not realize it.  

Fortunately, when resque starts a job it forks a child process and places the time it began the job in the process description.  You can tell if your worker is stale by using this timestamp to compare it to the current time.  Epoch Converter is one of a number of websites to help you convert the timestamp manually.

If you’re using god to monitor your processes, resque provides an example to eliminate these “stale” workers.  I don’t use god and instead use monit to monitor my processes.  Resque comes with a monit example too, but it only handles workers that consume too much memory.  If you happen to find yourself with stale workers, try using the following bash script I made.

#!/bin/bash 

# Simple script to kill stale resque workers
# Adam St. John [astjohn] [@] [gmail]
# 2011-06-24

# Timeout set to 10 minutes
TIMEOUT=600

# 21028 resque-1.16.1: Forked 17674 at 1308879414 
output=$(ps -e -o pid,command | grep -e [r]esque.*Forked | awk '{printf "%i#%i#%i\n", $1, $6, $4}')
now=`date +%s`

echo "`date` - Looking for stale workers."
# output should produce something like: 21028#1308879414#17674
if [ -z "$output" ] ; then
  echo "Stale workers were not found."
else
  echo "$output" | while read -r line ; do
  
    pid=$(echo $line | cut -d\# -f1)
    ftime=$(echo $line | cut -d\# -f2)
    fpid=$(echo $line | cut -d\# -f3)

    if ! [[ "$pid" =~ ^[0-9]+$ ]] ; then
      exec >&2; echo "Unable to find PID of stale resque worker"; exit 1
    fi

    if ! [[ "$ftime" =~ ^[0-9]+$ ]] ; then
      exec >&2; echo "Unable to find Forked time for stale resque worker"; exit 1
    fi

    difference=$(($now - $ftime))
    if [ "$difference" -ge "$TIMEOUT" ] ; then
      echo "Found a stale worker!"
      kill -s USR1 $pid
      echo "Kill signal USR1 sent to worker $pid so that it will shut down forked child $fpid"
    fi  
  done
  echo "Finished dealing with stale workers."
fi
exit 0

The script checks the timestamp that each worker leaves and kills a worker that has exceeded the TIMEOUT value. I run this script from a cron job.

Rails 3.1 Engines, Rspec, and Cucumber

I thought I’d make a quick post about my first encounter with Rails 3.1 and with engines in general.  Rails engines are essentially applications or a subset of functionality that can be run in or shared with other applications.  In Rails 3.1, creating an engine is as easy as:
rails plugin new engine_name [options]

where several options are available, including full and mountable. Run with
--help to find out more.

Generating a new engine produces a file structure similar to a regular rails application, but with a few notable differences which have been explained in several great articles.

Setting up Rspec

I enjoy using rspec in my testing and instead of using the test folder which was generated automatically, I decided to rename it to spec. Inside this spec folder, I set up the usual rspec environment. For example, inside my spec folder, I have additional folders named models, controllers, views, support, etc. Rspec needs to load the dummy application’s environment in order to function. To allow it to do so, a few changes are necessary to spec_helper.rb.

# /spec/spec_helper.rb

# This file is copied to spec/ when you
# run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../dummy/config/environment", __FILE__)
require 'rspec/rails'

ENGINE_RAILS_ROOT = File.join(File.dirname(__FILE__), '../')

# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
#Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
Dir[File.join(ENGINE_RAILS_ROOT, "spec/support/**/*.rb")].each {|f| require f }

RSpec.configure do |config|
  # == Mock Framework
  #
  # If you prefer to use mocha, flexmock or RR, uncomment
  # the appropriate line:
  #
  # config.mock_with :mocha
  # config.mock_with :flexmock
  # config.mock_with :rr
  config.mock_with :rspec

  # Remove this line if you're not using ActiveRecord
  # or ActiveRecord fixtures
  config.fixture_path = "#{::Rails.root}/spec/fixtures"

  # If you're not using ActiveRecord, or you'd prefer not to run
  # each of your examples within a transaction, remove
  # the following line or assign false instead of true.
  config.use_transactional_fixtures = true

  config.include Cornerstone::Engine.routes.url_helpers
end

Of course, renaming the test folder to spec did have some unintended consequences. For one, running rake tasks did not work. In order to fix that, I had to change Rakefile to load the dummy application’s environment from the new location.

# /Rakefile

#!/usr/bin/env rake
begin
  require 'bundler/setup'
rescue LoadError
  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
end
begin
  require 'rdoc/task'
rescue LoadError
  require 'rdoc/rdoc'
  require 'rake/rdoctask'
  RDoc::Task = Rake::RDocTask
end

RDoc::Task.new(:rdoc) do |rdoc|
  rdoc.rdoc_dir = 'rdoc'
  rdoc.title    = 'EngineName'
  rdoc.options << '--line-numbers' << '--inline-source'
  rdoc.rdoc_files.include('README.rdoc')
  rdoc.rdoc_files.include('lib/**/*.rb')
end

# notice the path change in the following line
APP_RAKEFILE = File.expand_path("../spec/dummy/Rakefile", __FILE__)
load 'rails/tasks/engine.rake'

require 'rake/testtask'

Rake::TestTask.new(:test) do |t|
  t.libs << 'lib'
  t.libs << 'test'
  t.pattern = 'test/**/*_test.rb'
  t.verbose = false
end

task :default => :test

Bundler::GemHelper.install_tasks

Finally, running rails on the console will not work until you change the dummy application’s environment reference in the rails script.

# /script/rails

#!/usr/bin/env ruby
#!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.

ENGINE_PATH = File.expand_path('../..',  __FILE__)
# notice the path change on the following line
load File.expand_path('../../spec/dummy/script/rails',  __FILE__)

I believe that’s all it took to get rspec working again after renaming the test folder to spec. Of course, since I’m using rspec, I no longer needed test_helper.rb and so I removed it. If there’s something I missed, or if you can’t get rspec to work, please let me know.

Preparing Cucumber

As with Rspec, Cucumber also needs to load the dummy application’s environment in order to function. After adding cucumber-rails to your gemspec or Gemfile, running rails generate cucumber:install will generate the necessary files and folder structure for cucumber to function properly. Unfortunately, the files generated are geared toward normal rails applications and not rails engines. To make it work for a rails engine, I had to make the following changes to env.rb.

# /features/support/env.rb

# IMPORTANT: This file is generated by cucumber-rails - edit at your own peril.
# It is recommended to regenerate this file in the future when you upgrade to a
# newer version of cucumber-rails. Consider adding your own code to a new file
# instead of editing this one. Cucumber will automatically load all features/**/*.rb
# files.

ENV["RAILS_ENV"] = "test" # had to explicity state the test environment (or cucumber env.)
ENV["RAILS_ROOT"] = File.expand_path(File.dirname(__FILE__) + '/../../spec/dummy/')
require File.expand_path(File.dirname(__FILE__) + '/../../spec/dummy/config/environment')

require 'cucumber/rails'

# If you use factory girl, I had to add the following...
require 'factory_girl_rails'
require File.expand_path(File.dirname(__FILE__) + '/../../spec/support/factories')

# Capybara defaults to XPath selectors rather than Webrat's default of CSS3. In
# order to ease the transition to Capybara we set the default here. If you'd
# prefer to use XPath just remove this line and adjust any selectors in your
# steps to use the XPath syntax.
Capybara.default_selector = :css

# By default, any exception happening in your Rails application will bubble up
# to Cucumber so that your scenario will fail. This is a different from how
# your application behaves in the production environment, where an error page will
# be rendered instead.
#
# Sometimes we want to override this default behaviour and allow Rails to rescue
# exceptions and display an error page (just like when the app is running in production).
# Typical scenarios where you want to do this is when you test your error pages.
# There are two ways to allow Rails to rescue exceptions:
#
# 1) Tag your scenario (or feature) with @allow-rescue
#
# 2) Set the value below to true. Beware that doing this globally is not
# recommended as it will mask a lot of errors for you!
#
ActionController::Base.allow_rescue = false

# Remove/comment out the lines below if your app doesn't have a database.
# For some databases (like MongoDB and CouchDB) you may need to use :truncation instead.
begin
  DatabaseCleaner.strategy = :transaction
rescue NameError
  raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it."
end

# You may also want to configure DatabaseCleaner to use different strategies for certain features and scenarios.
# See the DatabaseCleaner documentation for details. Example:
#
#   Before('@no-txn,@selenium,@culerity,@celerity,@javascript') do
#     DatabaseCleaner.strategy = :truncation, {:except => %w[widgets]}
#   end
#
#   Before('~@no-txn', '~@selenium', '~@culerity', '~@celerity', '~@javascript') do
#     DatabaseCleaner.strategy = :transaction
#   end
#

Note the two lines at the top which reference the dummy application. These lines are required because cucumber attempts to boot a rails application from the normal environment.rb which is usually located in the config folder. In a rails engine, this file doesn’t exist. It only exists within the dummy application. Adding the additional lines at the top of env.rb forces cucumber to bootstrap the dummy application’s environment. With these changes, cucumber should be fully functional.

One further change that had me stumped for a while was using Cucumber’s path_to helper located in paths.rb. I just could not make cucumber return the correct path when referencing using a path helper such as some_controller_method_path. In order to use these helpers (thanks Robin – see below) you can add the engines’ url_helpers to the World object of Cucumber like this:

module EngineRoutesHelper
  include MyEngine::Engine.routes_url_helper
end
World(EngineRoutesHelper)

Now, you will be able to rewrite the path_to method like this:

def path_to(page_name)
  case page_name
    when /^the discussions page$/
      discussions_path
    # etc…
  end
end

I hope my stumblings have helped others out there. Give me a shout if you run into any problems.

Converting a MySQL database to PostgreSQL using Rails

I recently decided to move one of my projects from MySQL to PostgreSQL for a number of reasons. I was looking for the easiest and quickest way to migrate the data. However, after playing with a number of solutions and tools, nothing was working well. Then I found this lovely rake task. The task uses rails to convert the data.

To convert your data, you will need to set up two databases. The rake task is designed to move data from the production database to the development database. To kick start things, download a backup of your production database and move it to your local computer. Next, establish a production MySQL database and then build the schema. rake db:schema:load RAILS_ENV=production. After the database is built, import the data to the local production database. mysql -u USER -p -D data_prod < backup.sql

Now, because the rake task moves data from a production database to a development database, you must configure your project’s database.yml file. This is what it could look like:

production:
  adapter: mysql2
  encoding: utf8
  username: rails_dev
  password: password
  host: localhost
  database: db_prod

development:
  adapter: postgresql
  encoding: utf8
  username: rails_dev
  password: password
  host: localhost
  database: db_dev

Note that the development database is set up for PostgreSQL. Therefore, before running the rake task you must establish this database as well. Build the database in PostgreSQL and then load the schema as above: rake db:schema:load RAILS_ENV=development. Also, do not forget to add a PostgreSQL adapter to your project’s Gemfile. I prefer the pg gem: gem "pg"

With the databases all set up, converting your data to the PostgreSQL database is as simple as:

rake db:convert:prod2dev

This worked great using ruby 1.9.2 and rails 3.0.8.

Rails 3.1 Engines – Mountable or Full? – Part 2

This is a continuation of my post Rails 3.1 Engines – Mountable or Full? – Part 1 which highlights some key features of a full engine. There are a few differences with a mountable engine, which I will try to explain below.

Mountable Engines

Engine Details

To generate a mountable engine, run rails plugin new MyEngine --mountable on the console. This will generate the engine’s basic structure.

The first difference I noted with a mountable engine is that, by default, the engine is namespaced. The engine’s starting config is:

module MyEngine
# my_engine/lib/my_engine/engine.rb

  class Engine < Rails::Engine
    isolate_namespace MyEngine
  end
end

Since the engine has an isolated namespace, the controllers, models, helpers, etc. are all bundled within the module MyEngine. The folder structure reflects this change as well. Unlike a full engine, the mountable engine generated the typical assets that are normally associated with a Rails application, namely application.js, application.css, and application_controller.rb. Note the namespacing that occurs with the application controller:

# my_engine/app/controllers/my_engine/application_controller.rb

module MyEngine
  class ApplicationController < ActionController::Base
  end
end

This namespacing occurs throughout the engine. There are subtle differences in terms of routing as well. With a mountable engine, the engine’s route file looks like this:

# my_engine/config/routes.rb

MyEngine::Engine.routes.draw do
  # route stuff goes here
end

Remember, the full application’s routes start with Rails.application.routes.draw. The dummy application’s routes file was generated as:

# my_engine/test/dummy/config/routes.rb

Rails.application.routes.draw do

  mount MyEngine::Engine => "/my_engine"
end

Notice that the engine is bundled under a “mounted” route at /my_engine. This means that all of the engine’s functionality will be located under the engine’s root location of /my_engine within the dummy (parent) application. This type of structure suggests that a mountable engine is best suited for situations when a developer wants the engine to behave as its own application operating in parallel with the parent application. Most of the engine’s features are isolated and independent of the parent application. To test this, I plugged the mountable engine into a test application.

Routing

I used a model within the engine called “Post”, along with the migration, controller, and helper to match (rails generate scaffold post title:string body:text). Now my engine’s routes file looks like this:

# my_engine/config/routes.rb

MyEngine::Engine.routes.draw do
  resources :posts
end

With a full engine, the Post routes were included directly into the parent application automatically. This time, using the mountable engine and running rake routes within the parent application, nothing happened! That’s because I did not mount the engine in a similar manner to what the dummy application in our engine is using. The parent application’s route file had to be adjusted like so:

# parent_app/config/routes.rb

ParentApp::Application.routes.draw do
 mount MyEngine::Engine => "/my_engine"
end

rake routes now produced the following:

my_engine  /my_engine {:to=>MyEngine::Engine}

…and that’s it. All of the engine’s functionality is bundled into the parent application underneath the route /my_engine.

After running rake my_engine:install:migrations and rake db:migrate in the parent application, I was ready to test how the engine’s controllers and helpers would integrate. [note: running migrations from an isolated engine will prefix your database tables with the engine name]. As with the full engine, I established a simple view template:

# my_engine/app/views/my_engine/posts/index.html.erb

<p>Hi There!</p>

Of course, navigating to /my_engine/posts in the parent application showed an almost blank page with the words “Hi There!” The controller actions and views from the engine were incorporated into the parent application, but only under the namespaced route /my_engine.

Next, I tested if the view could be overridden by the parent application. I created a new view template for the Post index action within the parent application that looked like this:

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>

No change and of course not! Our engine is namespaced, remember? Instead, I moved the template to parent_app/app/views/my_engine/posts/index.html.erb. Now visiting /my_engine/posts in the parent application showed “Good Bye!” That’s better. Next I decided to test out the engine’s helpers. I added a method to the Post helper inside the engine.

# my_engine/app/helpers/my_engine/posts_helper.rb

module MyEngine
  module PostsHelper
    def test
      raw("<p>hello world</p>")
    end
  end
end

I then changed the parent application’s view template to use the helper.

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>
<%= test %>

Visiting the page resulted in the words “Good Bye!” and “hello world.” Therefore, the engine’s helper methods were directly exposed to the parent application, allowing the parent to use them. 

 To be honest, I did not expect this to work. There is a section in the Rails comments about sharing an isolated engine’s helpers with the parent application by using helper MyEngine::Engine.helpers in the application controller of the parent. It states that in doing so “[the parent application] will include all of the helpers from engine’s directory.” Perhaps this is a bug in the release candidates of Rails 3.1? Maybe my interpretation of the instructions is way off? I’m not sure. If this does change by the time you use it, and you can’t access the engine’s helpers, I suggest trying the method described.

Summary

It seems as though mountable engines are best suited as complete applications in themselves which run along side a parent application. The engine routes, controllers, models, helpers, etc. are namespaced and bundled into the mounted route within the parent application, thus avoiding namespace conflicts. If I’m way off in my assessment or there are other things which I have missed, drop me a line and I’ll add them!

Rails 3.1 Engines – Mountable or Full? – Part 1

I decided to create my first rails engine about a week or so ago.  I figured this was a great way to learn Rails 3.1 and the new engine generator; however, I quickly ran into a problem.  Do I want a mountable or full engine?  What’s the difference anyway?  At the time, a quick google search showed that not many people really knew. The Rails code comments are great, but after reading them, I was still a bit confused.  For those of you out there in the same boat, here’s my attempt at highlighting some of the differences.

Full Engines

Engine Details

To generate a full engine, run rails plugin new MyEngine --full on the console. This will generate the engine’s basic structure.

The first thing I noticed was that the typical assets associated with a normal Rails application were not generated; application.js and application.css were not created. In fact, the application controller was completely omitted, alluding that perhaps a full engine is feared toward supplementing a parent application only. As far as routing goes, the engine’s route looked like this:

# my_engine/config/routes.rb

Rails.application.routes.draw do
end

… and the dummy application’s routes file looked like this:

# my_engine/test/dummy/config/routes.rb

Dummy::Application.routes.draw
  # route stuff
end

Another item that was missing from the basic application structure was the database folder. For engine configuration, a full engine is not namespaced, so the engine.rb file was essentially blank.

# my_engine/lib/my_engine/engine.rb

module MyEngine
  class Engine < Rails::Engine
  end
end

Now, the rails comments state that engines allow you to add a “subset of functionality to an existing Rails application.” Since the full engine is missing an application controller, it seems that its purpose is to plug into an existing application without namespacing to add functionality to the parent application. To test this theory, I generated a new application and referenced my engine in the parent application’s Gemfile.

Routing

Since the engine is not namespaced, the engine’s routes are incorporated into the parent application’s routes directly. To test this, I created a model within the engine called “Post”, along with the migration, controller, and helper to match. To create the migration, I also had to build the my_engine/db/migrate/ folder, which was not created when I generated the engine. Running rake routes within the parent application resulted in the following:

    posts GET    /posts(.:format)          {:action=>"index", :controller=>"posts"}
          POST   /posts(.:format)          {:action=>"create", :controller=>"posts"}
 new_post GET    /posts/new(.:format)      {:action=>"new", :controller=>"posts"}
edit_post GET    /posts/:id/edit(.:format) {:action=>"edit", :controller=>"posts"}
     post GET    /posts/:id(.:format)      {:action=>"show", :controller=>"posts"}
          PUT    /posts/:id(.:format)      {:action=>"update", :controller=>"posts"}
          DELETE /posts/:id(.:format)      {:action=>"destroy", :controller=>"posts"}

As you can see, the Post routes from the engine were included directly into the parent application.

After running rake my_engine:install:migrations and rake db:migrate in the parent application, I was ready to test how the engine’s controllers and helpers would integrate. In my engine, I had a simple view template:

# my_engine/app/views/posts/index.html.erb

<p>Hi There!</p>

Of course, navigating to /posts in the parent application showed an almost blank page with the words “Hi There!” The controller actions and views from the engine were incorporated into the parent application.

One helpful feature of engines is the fact that their controller methods and views can be overridden simply by placing similar files within the parent application. For example, I created a new view template for the Post index action within the parent application that looked like this:

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>

Now visiting /posts in the parent application shows “Good Bye!” That’s great, but what about the engine’s helpers? I tested that functionality by adding a method to the Post helper back inside the engine.

# my_engine/app/helpers/posts_helper.rb

module PostsHelper

  def test
    raw("<p>hello world</p>")
  end

end

I then changed the parent application’s view template to use the helper.

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>
<%= test %>

This time, visiting the page again in the parent application resulted in the words “Good Bye!” and “hello world.” Therefore, the engine’s helper methods were directly exposed to the parent application, allowing the parent to use them.

Summary

In summary, it seems as though full engines are best suited to augment a parent application. The engine routes, controllers, models, helpers, etc. are exposed to the parent application which allows for easy access, but could result in namespace conflicts.

It looks like this post is getting a bit lengthy so I’ll stop it here and talk about mountable engines soon.

San Francisco, USA

We arrived in San Francisco late in the evening, and the first taste of North America we received was the customs official yelling at people to fill the immigration lines. There was no please, or thank-you, and he was harsh and loud. We quickly figured out how to take the Bay Area Rapid Transit (BART), and took it to downtown. The BART system was much older, louder and dirtier than what we were used to. It was also run by an actual person, and not by a computer. At one point the conductor came on the loudspeaker and started yelling at someone standing in the doorway of one of the cars. This was a big contrast to the ultra modern system in Hong Kong or Singapore, where everything was fully automated, smooth, silent, and which also used the wind generated from movement for ventilation. The San Francisco subway lines were stuffy and hot.

Continue reading

Hong Kong, China

Well, it’s been a while since we’ve been back, but we never actually finished the story. Here are the last two entries.

Hong Kong is a city composed of many islands, mainly Hong Kong Island, and Kowloon Island. It is a financial and architectural metropolis located just off the East cost of mainland China. We left Singapore early in the afternoon, and arrived in Hong Kong in the evening, just after dusk. As soon as we arrived at the airport, we picked up our bags, and purchased an airport express ticket and an “octopus” card. The octopus card is a multi-purpose card that can be used for almost all forms of public transport including the ultra modern MTR subway, buses, tram cars, as well as at stores such as 7-11 and KFC. It only cost 7HKD (1 USD) with a 50 HKD refundable deposit. If more money was put on the card and not used before returning the card, there is a full refund issued. The airport express ticket was good for a single trip into the city on a modern subway/train with few stops.

Continue reading

Kanchanaburi, Thailand and toward Hong Kong, China

We pictured Kanchanaburi to be a small village along a river in the middle of some jungle, but it was actually quite a large city since it is the capital of its province. Although it’s a big city, it does have close and convenient access to national parks and waterfalls, but we were here mostly to visit the bridge on the river kwai and a tiger temple.

28733

The bridge on the river kwai, made famous by the movie, was basically the start of what became known as “The Death Railway.” The Japanese occupied Thailand and much of South East Asia during WWII, and decided that it was necessary to build a railway into Burma in order to occupy Burma, and cut off important supply routes to China. The railway was constructed by tens of thousands of POWs, many of which died from disease, malnutrition, fatigue, and the commanding Japanese themselves.

We stepped off the bus and were immediately greeted by the usual touts. This time we decided to listen to one of them; perhaps because we are at the end of our trip and don’t really care to hunt for a better bargain. The woman led us to a driver that was going to take us to the guest house for 80 baht each. When we started to walk away, he eventually lowered his price to 40 baht total. He took us to what turned out to be a really nice guest house called “Jolly Frog.” It looked like this place would be quite popular in the busy season since it had nice rooms, a nice courtyard with hammocks, and a massive restaurant with reasonable prices. After checking into our room, we decided to go for a walk towards the bridge.

Literally 15 minutes into the walk, we made a detour to an air conditioned internet cafe. The heat was unbearable. This was the first time in our trip that we had to stop walking, even in the shade, and get out of the heat. We stayed in the cafe surfing the web for at least two hours until it was around 3pm and the temperature had started to cool down. At this point, we were able to finish the walk, which turned out to only be a few minutes more to the bridge. Along the way, we noticed that a woman lost her hat as she drove by on her friend’s motorbike. An old man driving the opposite way slowed down and picked up the hat. Both of us were thinking that it was nice to see such generosity by the old man. However, instead of turning around and giving the hat back, he simply dove away slowly as if he would stop if someone complained. No one complained and he simply puttered down the road after stealing the hat. We still joke about the “nice old man.”

Continue reading

Ayutthaya and Lopburi, Thailand

We arrived late Friday evening in Chiang Mai and were too late to pick up our Buddha statue from the Fine Arts Department. In order to kill time we decided to visit the zoo on Saturday. We rented a motorbike and drove through some serious traffic to reach the zoo which was only about 20 minutes from the city centre. The zoo was pretty small, but somehow we managed to spend the entire afternoon there watching monkeys, lions, tigers, and giant pandas. On Sunday we slept in and strolled through the city for the day. In the evening, the Sunday Night market was set-up along a long walking street. The market stretched for about a kilometer along a cobbled stone street with vendors along either side, some even spilling into side streets and open wats. Buskers in the middle of the street kept everything entertaining, and there were plenty of food stalls. There was even a live performance of dancers and breakdancers. Although the market was very touristy, it seemed as though a large number of local people frequented it as well. It was easily one of the best market experiences out of the hundreds that we have had.

286792868228676

Continue reading