Netflix has open sourced one of it’s most powerful security tools, Scumblr. This post will go into the many uses of this for online marketers and webmasters, how to set it up from start to finish, and how to replace your existing brand mention monitoring services with this free tool.
What Is Scumblr by Netflix?
Netflix has an ongoing problem, people like to hack peoples accounts then post the login credentials onto forums, on pastebin, or on social media. Likewise, sometimes nefarious individuals may plan a DDoS attack on Netflix and will meet in some forum somewhere on the internet in order to coordinate it prior to being executed. Therefore, Netflix realized that they needed to create a tool which would monitor the entire web for specific keywords/queries, log them in an easy to digest manner, and then allow their employees to take action on them.
From this need, Scumblr was born. They open sourced it in 2014 along with Sketchy and Workflowy which are add ons we will discuss in a moment.
How Internet Marketers can use this amazing tool for brand mention monitoring.
After learning about Scumblr in an article about how Ashley Madison was using the service to search for dumped credentials to their infamous dating site, I realized that this is something that may be worth looking into. Turns out, the ability to search the web for any specific keyword was something I was already paying for, as I imagine many of you reading this do in order to monitor brand mentions for your or your client’s company.
Scumblr can be configured to ping you every time someone mentions your company on social media or in a forum for example. Additionally, you can use it to monitor pastebin for email dumps of addresses for you or your employees.
Who this tutorial is for
This is for a novice level marketer, entrepreneur, small business owner or even individual who would like to monitor mentions of them or their brands on the internet. This is done using Linux (Ubuntu in particular), so some basic knowledge of linux is required, though I will write this assuming you know nothing about linux in order to lower the barrier of entry to those wishing to utilize this amazing tool.
Step 1: Get Linux (Ubuntu)
This is a linux based program, therefore step 1 is getting a copy of Ubuntu. You can do this in one of two ways:
- Install Ubuntu on your machine and dual boot between Windows and ubuntu (not recommended)
- Create a virtual machine on your windows desktop that runs Ubuntu so that you can play with this while having windows still running (basically it’s like a window where an ubuntu computer is running). (Preferred method)
For the purpose of this tutorial, we are going to use a virtual machine, as I believe that is the best way to do it, that way it runs all day and you can access it whenever you want just like you would any other program.
Installing The Virtual Machine
What virtual machine you decide to use is up to you. If you know how to porate things, I strongly recommend VMWare workstation. The free option is Oracle’s “Virtual Box”. I do not recommend this one if at all possible, it has issues with resolution size.
Once you have downloaded the VM app of your choosing, you need to install ubuntu on it. To do this, go to ubuntu’s download page, download the .iso file, then save that for use in a moment.
For this example, we will use VMware Workstation:
Click File>New Virtual Machine
When the installer opens, click “Typical” and next.
Select installer disc .iso and navigate to the folder you saved the Ubuntu iso to.
Follow the rest of the installer prompts, it’s pretty straightforward from this point. It will ask you what hardware you want to emulate, which jsut means how much disk space and how much ram should ubuntu take. Depending on how fast your computer is and how much storage space you have, you can make these anything you like. Put 2gb of ram and 10gb of space if you want to play it safe.
Once it’s installed and ready, you should have a nice new Ubuntu virtual machine ready for your disposal.
Step 2: Installing Scumblr
So I will assume you have never used Linux before and do this step by step:
From your new ubuntu desktop, press ctrl+alt+t. This will bring up a new terminal window. It should look like this:
Here are the rest of the steps directly from the Scumblr Wiki. Just copy and paste them one by one into the command line, and let it do it’s work between steps.
This section will walkthrough a basic setup for Scumblr on a base Ubuntu 14.04 system. This guide assumes you have an Ubuntu system setup and available to go.
From the command line:
sudo apt-get update sudo apt-get -y install git libxslt-dev libxml2-dev build-essential bison openssl zlib1g libxslt1.1 libssl-dev libxslt1-dev libxml2 libffi-dev libxslt-dev libpq-dev autoconf libc6-dev libreadline6-dev zlib1g-dev libtool libsqlite3-dev libcurl3 libmagickcore-dev ruby-build libmagickwand-dev imagemagick bundler
From the command line:
cd ~ git clone git://github.com/sstephenson/rbenv.git .rbenv echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc echo 'eval "$(rbenv init -)"' >> ~/.bashrc exec $SHELL git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc exec $SHELL rbenv install 2.0.0-p481 rbenv global 2.0.0-p481 ruby -v
Install Ruby on Rails
From the command line:
gem install bundler --no-ri --no-rdoc rbenv rehash gem install rails -v 4.0.9
Install Application Dependencies
sudo apt-get install redis-server gem install sidekiq rbenv rehash
From the command line:
git clone https://github.com/Netflix/Scumblr.git cd Scumblr bundle install rake db:create rake db:schema:load
Create an Admin User
From the command line from the Scumblr root folder:
In the console (puts your details in between the <> brackets:
user = User.new user.email = "<Valid email address>" user.password = "<Password>" user.password_confirmation = "<Password>" user.admin = true user.save
go to the Scumblr directory (cd ~, press enter, then cd Scumblr) and do cd config/initializers.
cd config/initializers nano scumblr.rb.sample
This will open up the file where you enter in all of your api credentials.
Important! This is a demo file, so before you can use it remove the “#” at the beginning of each section you fill out.
Once you are done, press Ctrl+x, then “y” for yes, and then when it asks you what to name the file, remove the “.sample” part at the end. It should read “scumblr.rb”
Scumblr allows you to search the following providers:
- Goto https://console.developers.google.com/project
- Go to “Apps & Auth” tab on the left hand panel. Then select API’s. Then click “Custom Search Engine”
- On the next screen choose enable this api.
- Click on “credentials” on the left hand panel now.
- Click “Api Key”
- Choose “Browser”
- Click “Create”, leave the text box field blank.
- Go back to “credentials” on the left hand side of the page. Listed now are your credentials. This will go into the field in scumblr.rb titled “Config.google_developer_key” in between the two ‘ brackets.
- Now go to https://cse.google.com/cse/all to set up your customer search engine.
- Click the “Add” button.
- Fill out your screen as follows
- No open your new custom search engine and select “Search entire web but emphasize included sites”
- Then click “Search Engine ID”, this will display your cx ID. Back in scumblr.rb this goes where it says “config.google_cx”
- config.google_application_name should be named “Scumblr”
- version number just put “1.0”
Facebook API Credentials
- Goto http://support.fiddlefly.com/widgets/gallery/how-to-acquire-facebook-api-credentials/ to elarn how to get your facebook credentials.
Twitter API Credentials
- Goto https://themepacific.com/how-to-generate-api-key-consumer-token-access-key-for-twitter-oauth/994/ to learn how to get Twitter API keys.
Once you have all your credentials registered in scumblr.rb, press ctrl+x and save.
Open up 3 command prompts by pressing ctr+t From the command line from the Scumblr root folder.
In each terminal, do “cd Scumblr” so that all of them are now in the Scumblr directory. Then run one of the following in this order:
redis-server & ../.rbenv/shims/bundle exec sidekiq -l log/sidekiq.log & ../.rbenv/shims/bundle exec rails s &
Once those three commands above are executed, open up firefox in Ubuntu and goto http://127.0.0.1:3000/. Enter the username and password you created earlier.
The workflow of Scumblr goes like this:
- Create a search (telling Scumblr what to look for and where)
- See the results of those searches.
Creating A Search For Brand Mentions
Go to the “Searches” tab at the top. Then click “new search”.
From here you enter:
- Name: The name of your search (for your reference only)
- Query: what keywords you want it to look for. Try ti with your brand’s name.
- Select where you want Scumblr to search (Google, Facebook etc).
- Tags: Add tags to organize if you want
- Description: Add a description if you want.
- If you choose Google, set the “max results” to 100 which is google’s maximum. Every 10 requests count against your API limit, which for free google accounts is 100. So you get 1,000 searches in total, but can pay google for more.
After you create your search, click the big red “Run Now” button. Within seconds, your results will show up on the “search” page.
What Marketers should use this for:
- Run a google search to find any mentions of your brand name. If someone writes about you, you will get a notification.
- Find mentions of your brand on social media.
- Stop those planning DDOS attacks by monitoring people using the term “DDOS” and your brand.
- Put in your email address to see if ti’s been dumped on pastebin or elsewhere.
By leveraging this tool, you are getting for free what many are paying for. A scraper that finds mentions of you across the web and aggregates them for easy viewing. Netflix also open sourced Sketchy and Workflowy to automatically take a screenshot of the pages it finds and to email you about it. Workflowy allows you to create workflows such as “received”, “working on it”, “completed” etc.
What other cool things are you planning on using Scumblr for with your brand?