Monkinetic Weblog

RT @C_NyaKundiH: A farmer in Uttar Pradesh released 3 sacks full of 40 snakes in a gvt office after the officials asked for bribe. https:…

RT @WeNeedFeminlsm: female body hair is so forbidden that in razor ads a woman will be shaving an already bald leg...what does that tell yo…

Rachael Worthington is working on getting her blog set up as a static site built by Hugo, a new entry to the static-site generator field written in Go. She's having a few issues so I thought I'd write about a recent site I set up with Hugo and how that site is getting built/deployed.

I recently set up a static website for my father at

. His site doesn't change very often, and ought to load fairly quickly, so I figured rather than mess around with Wordpress or Drupal, I'd try Hugo. I'm pretty compfortable with Git so the site's source code is hosted on Bitbucket (free private repos, yay). And while I can manage git from the command line without always losing my cool, I tend to use Github Desktop which acts much like any source code management app and keeps me from having to remember how to do partial-file commits.

I looked into several continuous integration services: TravisCI, CircleCI, and Codeship, because I wanted to get the site built and deployed whenever I pushed code into Bitbucket.

TravisCI is popular among the open-source set because they have a free plan for open source projects, the next plan up is $129/mo and that's just not worth it for a couple of sites. CircleCI looked good but does not appear to support builds for Go-based projects.

Codeship supports Go, and I found this article about deploying a Hugo site using the service: Hugo Deployment via Codeship. My setup for is similar.

I connected Codeship to my Bitbucket account, and crated a new project for the repo. I created two build pipelines (described in the article above, I won't re-iterate here) - one builds off of the master branch and deploys to the main site. The other build pipeline builds from a feature branch (feature/responsive, for a responsive update to the site) and deploys to a subdirectory of the main URL.

My main build pipeline uses a "Custom Script" build with the following script:

# get hugo
go get -v github.com/spf13/hugo
# build
hugo --verbose
# deploy via ftp
lftp -c "open -u $FTP_USERNAME,$FTP_PASSWORD $SERVER; set ssl:verify-certificate no; mirror -R ${HOME}/clone/_site ."

So, yeah.... FTP. The site is hosted on a cheap GoDaddy account that I don't run, so there you go.

Hope this helps out, Rachael, and good luck on your blog!

Yeah, I've watched it about 55 times now. See for yourself:

Naturally I'm squueeeeeeeee-ing every time I watch, it, but I also have a few thoughts:

The music, oh my goodness the music. It's spare and starts as something not immediately recognizable as one of the SW themes, but it's simply gorgeous and flows into the John Williams scores we know and love.

The dialog is haunting, but also feels a bit like a reading of the character motivation notes:

> "I'm no one."

Rey (with Luke's "I gotta get off this planet" look)

> "I was raised to do one thing... But I've got nothing to fight for."

Finn.

> "I will finish... what you started."

Kylo Ren (crushing on Vader)

It already feels like Star Wars, more than the prequels ever did. SO EXCITE.

My Wife is currently fuming in the other room because she sent the wrong signup link to several business partners, because someone editing that the link did so in an HTML/Rich Text editor, and ended up editing the text of the link, not the source. So she sees the correct link in her editor/email, but in fact she's sending someone else's affiliate link.

I pretty much only use plain text email, and this is one reason - if it's not spam or phishing, it's simple ignorance.

RT @BabyAnimalPics: when people waiting on your downfall but you the GOAT 🐐 http://t.co/loVvtxRdCS

Lightning talks, keynote by a PSF head, and a keynote by Jacob Kaplan-Moss. Jacob's keynote was excellent, focusing on the myth that programmers are either great or awful, and how that myth harms the industry by excluding anyone who doesn't fall into the stereotypical "programmer" temperment, skill-base, or even gender/race profile (we are not all Zuckerbergs).

I didn't really try and take notes, deciding instead to just listen and absorb. One point really stood out though: We will know we have succeeded when marginalized groups (women, persons of color, etc) are represented in our community not because they are exceptional developers, but because they are, just like all of us, appropriately adequate to the roles they fill.

Also, I'm re-reading @shanley's essay on the "10x Engineer". I'd recommend getting her collection of essays on startups and reading it; there's a lot of research and analysis there that Jacob reiterates in his talk.

My name is Steve, and I'm a mediocre programmer.

Beyond Grep: Practical Logging and Metrics, Hynek Schlawack

Agenda:

• Errors
• Metrics - what's going on?! Without SSHing
• Logging and centralizing it

Error capturing:

• fast notifications
• one notification
• useful context

Sentry (can you deploy it internally? Hynek says you can but I am not sure...). You get useful notifications with context, stack trace, and a "view on sentry" button, take you to see graph, number of occurences, etc. Get lots of metadata.

*Facepalm: it's right here

*

Pushing data in: raven-python.

Metrics

Numbers you save in a database! Numbers over time actually mean something. It helps you know, vs guessing.

Difference between server metrics (things you observe on a server), and application metrics (things you measure in your app).

• counters (how many times things happen)
• timers (how long things take)
• guages (number of active things)

Aggregation for trends, etc.

Correlations:

• freq/sec vs. latency, for example.

Math:

• No. reqs/second
• worst 0.01% request time
• don't try this alone! - use tools by those who know how to do this.

Monitoring:

You can apply monitors to your metrics - notify on latencies, error rates, or other anomalies - "tell me if something starts getting our of hand?"

Metrics really belong in a time series database. How long you store them and at what frequency can make a difference.

Three options:

• Librato Metrics (SaaS)
• Graphite/Statsd
• Grafana - pretty Dashboard that pulls from graphite or influx.
• InfluxDB - Graphite++ written in Go

Options for getting data into your timeseries DB:

1. External aggregation: StatsD, Reimann

   • no state, simple
   • no direct introspection

2. Aggregate in-app, deliver to DB

   • in-app dashboard, useful in db
   • have state in app, which is bad

For second option in-app, option is greplin/scales; collect data in the app and scales gives you a dashboard (using flask or the like)

Logging

Splunk: $$$ Enterprise $$$

Papertrail, Loggly: SaaS, seinding data to thrid party not fun

ELK: Elastic Search, Logstash, Kibana

Data - the goal:

@timestamp {
    json: data
    that: is
    readable: info
}

Structlog

Structlog is not a logging system, instead it wraps your logger, and adds context and other stuff.

Log to standard out, and let other features (log rotation, etc) be handled by the subsystems

Log Capture:

• files
• syslog
• pipe into a logging queue

Wait a minute...!

How do we put these three components together?

Errors:

• Capture errors in the logging integration
• Capture error info in a custom error view

Metrics:

• try to keep the metrics observable from outside (ie, metrics captured from the webserver)
• middleware (capture metrics around the request, not in it)
• extract metrics from logs (send from logstrash to graphite, for example)

Leverage monitoring:

• measure data and save it out (nagios can capture metrics from checkers)

Remaining:

1. measure code paths
2. expose guages (how many of x right now)

Serialization formats are not Toys

yaml.load is terrifying.

More notes later 'cause I'm playing with structlog.

Started my day with a massive ongoing migraine that had me hiding in my hotel room with a towel over my eyes, praying for either death or my migraine meds to start working.

Pythons are Deaf and so are some Pythonistas

Will write up my notes on this amazing session later once I can digest it. As a parent of deaf kids and someone interested in culturally appropriate user interfaces this session has been the conf highlight so far.

Debugging "Hard" Technical Problems, Alex Gaynor

What is debugging?

When does this bug occcur? (Figure out "Sometimes my server crashes" vs. "When this happen my server crashes")

How does it crash? In a particular situation, or in a particular function?

What is a "Hard" problem? It's something that takes more than 10-15 minutes to debug/figure out. Like, really unexpected. When a normal adhoc debugging process fails, now you're in "hard" territory.

Timing and ordering bugs: thread, greenlet, asyncio, etc. It's hard to track what's happening in different streams in your program. Also ordering bugs that make it hard to reason about the order of things happeniig.

Bugs that cross library or system boundaries. API docs and interface changes, issues.

The worst: many indepent "safe" failures that conspire to make a super bug.

Principles of Debugging a Hard Problem

Everything is in scope.

Normally you can assume that the OS, stdlib, etc are not at fault. But a tiny bug in a system could conspire with a small bug in your code to really fail.

Read the source. Read all the source

Read your own source, read the source for libraries you are using.

Trust Nothing

Don't trust repr() - Django querysets look identical to lists in repr().

Beware the monkeypatch. weeps

Get a lab notebook and document debugging steps. Use vcs branches as a lab notebook, using a branch for each test investigation.

Tools

• debugger for your language (pdb)
• OS tracing (dtrace) - tells your what syscalls your code makes (can be hidden in your code)
• Editor or IDE you can easily read code in
• grep
• Domain-specific visibility tools (lsof for open files, netstat for open network sockets, etc)

Also, application-specific tools. App metrics, logging, etc.

Techniques

Pair Debugging: Talk about what your trying to debug, show the process, talk about what you see in the output. Compare perceptions or assumptions about what is happening.

Minimization: Find a smaller program that fails in the similar way as a larger problem. if your program uses consurrency or randomness, try removing those aspects to simplify. Reduce the number of interactions you have so to reduce the complexity quadrilaterally.

Proximate Cause: Did you just cut a release? Did you just make a change? Use a tool like git bisect to find the change that introduced the bug.

Keep your eyes on the prize: If you find a random broken thing, don't fix that right away, document it and come back to it.

Get out of production ASAP: If the bug is in production, find a way to reproduce it outside of production right away.

Story Time!

Alex tells a story of a bug that took 3 people more than a day to debug, about 15 hours of debugging.

My second day at PyCon, and the start of the actual conference.

Breakfast, show floor walking. Conversation with Openstack vendor, I'm thinking we (at $DAYJOB) should commit to integrating with Oslo messaging (AMPQ) for events. The new event/messaging system is not coming any time soon.

Friday Keynote: Catherine Bracy, Code for America

> In the words of a former boss, "Technology will f*** you".

Catherine speaks about how problems with technology intended to help (Healthcare.gov, Florida unemployment site, Arizona welfare site) end up hurting the populations they are meant to serve. We feel like government doesn't work, when we expect to be able to access the store of the world's knowledge from a device in our hands. It is dangerous for democracy and breeds cynicism when digital interactions with our government don't work.

In 2015 that means that goverment must be able to do digital services well. There is no Code Red team to come save us when things go bad.

Stop Waiting for Permission - Medium post: Code for America hackathon particpant who built new tools to help people know what the parking rules are in different parts of Philly.

Detroit Water Project - CFA Fellow built a system to enable people to assist others who need help paying their water bills in Detroit. Recently graduated Y Combinator.

Local Free Web - find free internet access via text message based on bus stops.

Government can work in 21st century, but it means people who have the skills contributing to the tools and processes that need improvement.

Misc Notes:

• Code for America focuses on the improving the bureaucracy of government, not on the politics.
• Focuses on implementation and trasparenncy rather than policy.

Building Secure Systems, @lvh

The problem:

• All software has bugs
• Some of those bugs are security bugs
• All sec bugs are bad

Bad news:

• Tools don't work
• Corporations mess up security
• The education system doesn't educate

Tools: How do you write a test for "security"?

How do we learn software?

Read the docs, but sec docs are missing or awful. lots of misinformation. Good practice looks like bad practice.

How else do you learn: you play with it. Lather rinse repeat. but sec does not work that way. Install, play with it, get it wrong, repeat.

How else can you learn?

Install, try to break it, learn, repeat. Failure is harder, there are more failure states, issues are more subtle.

Lots of companies mess it up: no time for security testing, people make mistakes. Good rpactices look like bad practices.

What do we teach/not teach?

• Liskov Substitution: A class can be replaced by a subclass
• What don't we teach
  • Principle of Least Authority: don't give a system more power than it needs.

Education can be fixed

• Code review
  • disseminate experience
  • best thing after bug tracking
• adversarial approach
  • in-house (red v blue)
  • don't stop til it doesn't break
• automated
  • certificate checking
  • middleware tests

Meta Conversations

At this point I couldn't keep up anymore on the slides - the presenter was flying through them and I did the best capturing the information - look for them online.

I'm asking myself what in-house apps need security-wise. How best to implement Business-Need-To-Know?

Smart Services: Microservices, Frank Stratton, Runscope

What technologies, how things are organized. Microservices are highly indenpendent codebases. Lots of moving parts though, so requires orchestration. You have to invest in infrastructure to use microservices.

Microservices == SOA + DevOps.

Example:

• Start with dashboard, wanted to build new service: identity
• Each service is on a separate host
• First step is usually:
  • http request from one service to another

But, what happens when hosts are added or change? How do you handle multiple environments?

How do you find a service?

Runscope created Smart client: a wrapper around Requests.

service://identity/<service> -- this is a custom service URL that is translated at runtime by the client.

How does it find mappings for services? With another service, Atlasm which keeps records of mappings of hosts and services in DynamoDB.

On each service host: Sidecar, a service that runs haproxy that maps localhost service calls to the remote service. When services cahnge, haproxy is updated. Sidecar uses haproxy ACL to map services to new hosts/clusters.

Services all end up call out to localhost, which is then mapped via the HAProxy ACL to the appropriate service.

How do you make a service?

Smart service: Addons to Flask/Restful for building services. For example Runscope/healthcheck` adds healthchecks and environment data. Smart service setup returns a wrapped Flask app (like other flask addons)

Smart Config: loads configs for service -- can query Atlas at runtime for base config.

current_app.name
current_app.realm  # dev, test, production, etc

current_app.redis  # redis client
current_app.db     # db client
current_app.etc    # ...

Also:

• common logging config (/var/log/runscope/etc)
• cli tool that generates a common app skeleton (sets up git repo, virtualenv, file structure)

These smart setups reduce cognitive overhead. Lots of "now I need to setup X" goes away.

Now what?

Automate everything. Puppet/etc to automate config deploy and setup. Setup monitoring, etc.

No shared databases

No shared dependencies (schemas, etc).

• Service with API
  • owned db
  • owned endpoint and load balancers
  • owned redis, etc

Makes deploying code really easy, because you are only deploying one service, not lots of other dependent code/systems.

To check out:

• Runscope/healthcheck - I already have several services this would benefit.
• HTTPbin
• RequestBin

Software Relationships! Don't get sucked into the drama

• Deliberate Ignorance
  • Use public APIs
• Evaluate your options
  • Evaluate service dependencies
  • Evaluate library dependencies
  • Does it have API guarantees?
• Wrap difficult depencies in your own layer to protect yourself later.
  • IE, Requests wraps httplib

Protect others from your drama:

• Make sure your library/code is understandable
  • question your own assumptions
  • ignore your own internal knowledge ("oh, everybody knows X")

At this point I have a headache from using my glasses for the computer and looking over the top of them to view the speakers. Sigh.

Your System's "Identity"

• "utility" or "solution"
• "transparent" (users know about the things under the hood) or "encapsulated" (users don't need domain expertise)``

[Ed. ELK stack is definitely a transparent system.]

Wrap up

About now I'm still jet-lagging, and decided that a good rest would make me right with the world for the more advanced session I want to attend tomorrow, so it's back to the Reine Elizabeth to snooze, go over my notes, and get some dinner later.

Dear April 10, 2015

I first learned John Harris' name from personal friend John Scalzi [Ed. John Scalzi is not a personal friend of the author], whose excellent series of books in the Old Man's War series are graced with Harris' atmospheric and inspirational paintings.

Perusing a sampling of his fine art prints or a Google image search, however, yields tens if not hundreds of familiar scenes, covers of largely science fiction books over the years.

> a selection of reasonably-priced prints which include images which act as a reminder of that writing. Isaac Asimov, James Blish, Frederik Pohl, Orson Scott Card, Samuel R Delany, Ben Bova, Allen Steele, and many more

I am in love with his art and seeing his work, or similar, on a volume in the local library will at least guarantee a look from me. John (Harris, not Scalzi) has a range of prints available (linked above) of his cover art, as well as of his poster work for Sinclair in the 80s, and (if you're into boats) a lovely series of pastel illustrations of racing yachts. Any sci-fi or art lover should definitely check out his work.

Update:

I didn't realize that there was a book of John (Harris') work, with an intro by John (Scalzi). While neither frameable or mountable on a wall, this tome is considerably cheaper than the 6 prints I want (wah) and would make a lovely gift for your humble blogger...

Today was my first day of PyCon, in Montréal, Quebec (my first). I'm headquartered with two other Wells Fargo teammates in the Fairmont Queen Elizabeth (Fairmont de la Reine Elizabeth), a 10-minute walk or so from the Palais des Congrés, where this year's PyCon is being held. It's an impressive building, though in the end, one tutorial room and conference hallway look pretty much like another.

Timezone changes (west-to-east sucks) had me up later than I wanted last night, and up earlier than I wanted this morning, but I walked over to the Palais in the brisk 40°F weather to get registered. One coffee later I found myself in the room for the first tutorial session I had registered for, only to find that I had not actually registered. The staff person managing the room was kind enough to let me stand in the back after all the sold-out seats had been filled, so I was able to at least listen and follow along with the notes.

This session was Descriptors and Metaclasses, and dug much more deeply into these advanced topics than I had before. My daily work digging through Horizon's internals made more sense especially after the presenter -- Python Academy's Mike Müller, who did an excellent job -- took us on a tour through the way that Python metaclasses work. My homework tonight while I de-jetlag is to work through the incredibly detailed tutorial notes in iPython notebook and make sure I understand them.

More rest tonight, things begin in earnest tomrrow!

AMC's and HBO's hits are all the rage for the hipsters on the Twitters, but Jodi and I prefer to curl up with a nice English cozy.

> Cozy mysteries, also referred to simply as "cozies", are a subgenre of crime fiction in which sex and violence are downplayed or treated humorously, and the crime and detection take place in a small, socially intimate community.

Some of the series with which we've passed comfy nights:

• Midsomer Murders: 11 seasons and counting - we did love us some Inspector Barnaby. :-)
• Rosemary and Thyme: Gardening sleuths, a perfect example of the genre.
• All the Morse/Lewis mysteries - Inspector Morse, Inspector Lewis (no longer on Netflix), and the Morse prequel series, Endeavor (iTunes).
• Miss Fisher's Mysteries: Phryne Fisher, Australian flapper and nosy, independently wealthy investigator (as many of the ameteur sleuths in these shows are :-) ).
• Poirot, naturallement.

At $DAYJOB I now manage between 15-20 python libraries that implement extensions to Openstack's Django-based dashboard, or are microservices that implement REST APIS (or both). For a while I've been working on adding documentation written for Sphinx in ReStructured Text. It's taken a learning curve but I'm slowly getting the toolchain and build process going.

The next thing I wanted to tackle was maintaining a readable changelog for my current project. This weekend I started experimenting with Jeff Forcier's Releases library. Releases is a Sphinx extension that makes it pretty easy to maintain the changelog for a python library (or anything you can write sphinx docs for). Jeff wrote Fabric, and Releases came out of his struggle with how to best maintain Fabric's changelog. Changelogs are tricky - trying to generate one from version control can be automated but doesn't give you control over how much information to provide about a given feature or fix, and forces commits to be written with the changelog in mind. Manually maintaining the changelog obviously gives you fine control over what to include and the language tone, but making sure that the log includes what bugs/fixes get into various releases is tedious. Jeff describes the the myriad issues involved in his original post on his solution to this thorny issue.

Enter Releases:

>Releases is a Sphinx extension designed to help you keep a source control friendly, merge friendly changelog file & turn it into useful, human readable HTML output. (from the Releases documentation)

Releases turns a file like changelog.rst into this (Fabric's changelog).

I decided to start testing Releases out, and ran into my first bug right off: the code I'm writing has not yet had a release, and writing a changelog like this (without an actual release listed) doesn't work:

=========
Changelog
=========

* :feature:`1` Implement REST service

    * Flask-based API server
    * Celery-based queue
    * Documentation for Installation and API

I chatted Jeff up on Twitter and then IRC, and we decided this was not really a bug, but an edge-case he didn't have, but was something that should be fixed. There's now a ticket filed on Github, and I did find a workaround: adding a no-date release at the bottom fixes it (and it does not render the null release):

=========
Changelog
=========

* :feature:`1` Implement REST service

    * Flask-based API server
    * Celery-based queue
    * Documentation for Installation and API

* :release:`0 <>`

RT @SciencePorn: This is the best headline you'll read today. http://t.co/DG8TZG0T4o

I'm not a big gamer. In fact, I'm not a gamer at all right now. But for about 8 months I played EVE Online, the decade-plus-old MMORG of internets spaceships, and I got hooked. I mean, Internet Spaceships?! I wandered around on my own for a while, got into a corp that got AWOXed, then found Rixx Javix and Stay Frosty, which became my adopted piratey home. YARR!

Even though I ran out of time and resources to play, I still follow along on Twitter and other social media and forums, especially the goings on in Stay Frosty.

This week is Fanfest, the huge gathering of EVE players and CCP Games, the company that makes EVE, all in Reykjavik, Iceland. Some of my ex-corpmates (including Rixx) are there, so it's fun to follow along and see what news comes next!

Stay Frosty!!

Well, here in Arizona it's been Spring for a month or so, even while the East Coast is still buried under record snow. Spring, usually means a renewed interest in my oft-neglected blog, and this time I realized that monkinetic had been down for some time and I didn't even know it.

I'm a very bad nerd.

Oh well. So to reduce the mental and technical overhead I've ported the damn thing back to Jekyll and moved it to Github Pages, where I don't have to futz around with hosting, and Fastly is still making it, well, fast. So let's see how 2015 goes in the blogging category.

RT @SallyStrange: Schrodinger's douchebag: A guy who says offensive things & decides whether he was joking based upon the reaction of peopl…

RT @bpoetz: programming is awesome if you like feeling dumb and then eventually feeling less dumb but then feeling dumb about something els…

@freebsdgirl no u dint #shotsfired