Instead, he worked in customer service for a financial services firm in Belfast, Northern Ireland. His favorite part of that job was helping stop fraud, though it was frustrating because people usually called customer service too late after they had sent money to scammers. 

Andersen heard about a risk analyst position at Signifyd, where he would be focused on fraud prevention. He jumped at the chance, joining in September 2019. Today, he is a front-end software engineer. 

Johannes Andersen

At Signifyd, moving from one team to another is not only possible – in Andersen’s case, his managers suggested it.

Andersen’s job as a risk analyst involved looking at the small percentage of online orders that Signifyd’s machine learning system had flagged as high risk or couldn’t validate. He’d verify whether the orders were legitimate. Risk analysts also look for trends or review patterns for specific customers.

“We get a lot of fraud trends,” Andersen says. A recent example: “tricking elderly people into thinking they’re in love.” Scammers prey on lonely seniors on dating sites, establishing a relationship and ultimately inventing a story about needing money or products available online to get out of a crisis. Risk analysts help define and catch new and growing trends to help the model improve to meet the challenges of the changing fraud scene.

Internal users benefit from a detailed interface

As a risk analyst, Andersen dealt with a lot of data: seller information, buyer information, discount codes, purchase amounts, and how long users spent on each web page. Not everything the analysts needed to see was displayed in their user interface – or if it was, it wasn’t always in a useful format. For example, it’s useful for risk analysts to see how long a user stayed on a page while making a purchase. If the purchase is fraudulent and was done by a bot, it may happen in just milliseconds.

However, the review interface did not show the information directly. It was hidden in JSON code in a separate tab, which made the numbers difficult to grasp quickly. To make matters worse, the times were all reported in milliseconds, which made the numbers difficult to grasp quickly. 

“The human brain doesn’t convert milliseconds to minutes easily,” Andersen said. 

The team requested upgrades to their interface from engineering – the same engineering team Andersen has now joined – but because that team also works on customer-facing tools, their requests sometimes took time to fill. 

So Andersen put his programming skills to work, creating a private Chrome extension that converted milliseconds into minutes and seconds and displayed the time-on-site information on the main console, making it easier for the analysts to quickly spot unusual transactions or patterns.

Software developer, heal thyself

“After work, I created tools to automate the risk analyst job I was doing day to day,” Andersen says. “I created UIs and buttons to make it a bit easier to get data.”

His managers noticed his interest in — and aptitude for — programming. They asked him about his career goals.

This is a common conversation at Signifyd, where all managers are given tools to help employees create development plans, says Annie Eller, senior HR business partner. “These include creating a leveling, or career ladder, showing the path for growth and then talking with employees about their career goals,” she says. “If an employee is interested in a career path in another department, we encourage the employee to talk with the manager from that other department to understand that department’s career path and what skills are needed to develop into that role.”

When Andersen’s managers asked him about his goals, he did not immediately say he wanted to join the engineering team. He didn’t have a clear career path mapped out for himself — he had worked in retail before getting the job in customer support. From there he moved to Signifyd. 

“I’d been programming for years, but I never wanted to do it as my job,” he says. “So I said I wanted to go into data science.”

His managers had seen his programming skills, though, and they thought he would make a good member of the front-end engineering team. They made him an offer: Try working on the front-end software engineering team for three months. If he didn’t like it, they would help him move to the data science team. 

Working with managers leads to a new opportunity

So in January 2020, he joined the front-end engineering team on a temporary assignment, working on the customer console that users see when they log in, as well as on tools used internally. By April, it was clear that he wanted to make the move. So, he joined the team and began work on a training program that he and his manager designed to ensure he’d have the skills and resources he needed to be successful.

The program included quarterly goals, such as joining the pager schedule to be on call for emergencies or helping onboard a new employee. He had one-on-one meetings with his manager and his group’s principal software engineer to go over his progress.

Andersen enjoyed the work so much that he completed three years’ worth of goals ahead of schedule, and in January 2021, he was promoted to a full-fledged software engineer. He is continuing to work toward his Amazon Web Services certification, which the company pays for. 

“When you know the basics of programming, you can learn pretty much anything else,” Andersen says. “You just need the general mindset of expanding on your knowledge.”

Another key to his successful move: being willing to try a position that the company needed to fill. “We had a lack of front-end engineers,” Andersen says. 

Moving to a new team is very possible for Signifyd employees, and “hunger on the part of the employee to understand what they want and to go for it” is one of the key ingredients, says Emily Mikailli, senior vice president of people operations. It’s great for employees to talk to their managers about their goals, but it’s also OK for employees to start working toward a new goal on their own. “Sometimes people want to explore things without looping in their manager first.”

Andersen says he is now working toward his next promotion, to senior software engineer. Working on the engineering team has also given him new ideas for long-term career goals: In a few years, he’d like to move into engineering management. 

Photo by Philippe Jausions on Unsplash

Looking for an organization that believes in growing together?

The post A software engineer develops a new career path appeared first on Signifyd.

I’ve worked at companies of all sizes in half a dozen different industries and every one of those companies has a different process for building and deploying applications. However, over the past five years, there’s been one phrase that’s been commonplace: “Just build a Docker container.” Of course there are a lot of steps that happen before and after building the Docker container, but Docker unifies build processes and makes application deployment significantly easier than it used to be.

On the surface, building a Docker container for your application is quite simple. I won’t go into the details here on how it’s done as I assume that you either already know how to do it or are willing to read one of the 54.3 million options on Google that are returned when you search “How do I build a Docker image?”

When you build a Docker image, Docker creates a series of layers for you, each one representing a cached step in the build process. When you push your Docker image, all of those layers (which don’t exist on the remote server) are pushed to that server. When someone pulls down that Docker image, they retrieve all of the layers of the image which don’t already exist on their machine. This saves a lot of time and space for many builds. However, it falls short in a few areas.

Secret values present a potential Docker security vulnerability

Part of your build process might require use of a secret value (such as a password). This is common when pulling artifacts from a private, remote server such as Nexus, Artifactory, AWS S3, or anywhere that you might keep Python packages, JARs, or really anything that only folks within your organization should have access to download. If you run a download step with some secret value using RUN as part of your build process, then that secret will be available in the cached layer to anyone who downloads your image. This is a security vulnerability.

For years now, folks have circumvented this problem by using multi-stage builds, essentially building two (or more) images within one Dockerfile. Suppose you need to use pipenv to download and install Python packages from a private, secure PYPI server. The first image is built with something like this:

FROM python:3.7 AS builder

followed by a:

RUN PIPENV_PYPI_MIRROR=somethingwithyoursecret@server pipenv install

Next, we specify our base image again with FROM and we build our image as we usually would, but instead of downloading the packages this time, we’ll just COPY them out of the previous image:

FROM python:3.7
COPY --from=builder /tmp/python_packages /opt/python_packages

This is great because the layers created by the first image are not cached, so the secret isn’t stored and isn’t pushed with our final image. However, this approach falls short because the layers created by the first image are not cached, so the next time we build this image, the whole thing will be rebuilt from scratch.

Luckily, Docker released version 18.09 (circa late 2018) with support for BuildKit which solves this problem. BuildKit makes a number of improvements to Docker, but most notably in this case, it allows us to mount the secret into the image, avoid multi-stage builds, and maximize Docker’s ability to cache layers.

Given our theoretical build process that requires connecting to our private PYPI server to install Python packages, you could update your build to use Docker BuildKit like so:

1. Add:

   # syntax = docker/dockerfile:1.0-experimental

   to the top of your Dockerfile

2. Put the full secret (in this case username:password@yourpypiserver) into a file on your machine. Call it something like:

   pypi_secret.txt

3. Update your build command to something like this:

   DOCKER_BUILDKIT=1 docker build --secret id=pipenv_pypi_mirror,src=pypi_secret.txt --output=plain

   (Note: BuildKit will try to collapse output in the terminal. The  –output=plain bit will stop this)

4. And then in your build, you’d just update your pipenv install command to this instead:

   RUN --mount=type=secret,id=pipenv_pypi_mirror PIPENV_PYPI_MIRROR=`cat /run/secrets/pipenv_pypi_mirror` pipenv install

The syntax is weird and I’ll admit that I have to look it up anytime I plan to use it, but the results are phenomenal. With this, you’ll get the standard caching that Docker provides without publishing your secrets to anyone who downloads your image.

Consider this work-around for a standard caching challenge

However, we still have a problem with regard to caching: if we make changes to our list of required Python packages, or if a build step prior to our pip install changes and invalidates layers in the cache, we have to re-download and re-install the Python packages again on the next build.

To get around this, we can specify a directory in the container that we would like to mount as a cache. This way, if the pip install step ever needs to run again, packages can be pulled from the cache instead of being pulled from a remote server. To do this, simply mount the directory with:

RUN --mount=type=cache,target=~/.cache/pip pip install -r requirements.txt

(Note: There’s a full list of other options available on GitHub.)

The first time you run your docker build command, you’ll see things like this:

Downloading numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)

If you change your `requirements.txt` file by adding or removing a package (such that the required version of numpy does not change) and re-run the build, the layer will technically be invalidated, but you’ll see output like this instead:

Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)

Docker BuildKit addresses security, caching and build performance concerns that avid Docker users have had with Docker since its inception. The technology is currently marked as experimental and isn’t supported yet by most build framework plugins. However, it provides a large number of benefits that make it worth using as part of your build process today, even if you have to roll your own plugin.

Looking to join a team of creative innovators? Let’s talk.

The post Securing your Docker build process without sacrificing speed using Docker BuildKit appeared first on Signifyd.

However, it is also one of the least understood roles thanks to the various ways it’s applied in Agile frameworks. Scrum is one of the most popular Agile frameworks and is the framework of choice for Signifyd. Within Scrum the product manager is known as a product owner (PO). When such a pivotal role is poorly understood and applied, it almost always leads to a product’s failure. It is often an easily preventable failure by avoiding common bad practices but every PO shouldn’t have to experience them first hand before they know to avoid them again.

What is a product owner?

The PO role has seen a surge in adoption in recent years — especially in the UK, thanks to the mainstream adoption of Scrum.

Each industry and company has various interpretations of the PO role. To ensure we are all on the same page, I’m sharing a definition of the role within the Scrum framework. The product owner is the voice of the customer, represents the internal stakeholders and is responsible for delivering the highest possible value to the business. 

The role carries a range of responsibilities between the two main internal stakeholders:

This wide-ranging array of responsibilities illustrates the complexity of the PO role. It’s easy to understand how multiple interpretations can arise.

The dreaded anti-patterns

The PO role is commonly poorly applied. Rarely done intentionally, the problem starts from a lack of understanding of the PO role itself, or in Scrum or Agile. This sets everyone up for failure: the individual, the team and the company.

This can lead to anti-patterns: common responses to recurring problems that are ineffective and counterproductive. Most people working in software engineering have encountered their own anti-patterns. I’m sharing a list of those I love to hate. However, rather than just rant about them, I will also provide some suggestions for addressing each one.

Anti-pattern No. 1: Serving multiple teams

This is a common issue when organizations scale quickly without the resources to match their growth. The PO can sometimes be a victim of their own success when they’re asked to rescue multiple struggling teams.

Problem

• The PO’s availability is stretched with reduced effectiveness.
• The PO becomes a bottleneck for decision making.
• Product ownership does not scale the same way as servant leadership (also known as scrum masters) potentially can, which can lead to inflated expectations.

Solution

• The PO should limit themselves to a maximum of two teams or risk major performance reduction.
• The PO should monitor their availability across their teams and address any issues early on to avoid negatively impacting their teams.

Anti-pattern No. 2: Multiple POs in a team

The inverse situation of one PO serving multiple teams is just as bad. A multiple PO set-up can happen when a team or company is transitioning from Scaled Agile Framework (a framework for scaling Agile principles in enterprises) to Scrum where roles and responsibilities differ. It can also occur when splitting the responsibilities across two roles via a product manager (strategic responsibilities) and product owner (tactical responsibilities) combination. The latter happens when an individual may not possess all of the required skill sets or in a surplus of resources.

Problem

• There’s no single source of truth.
• Teams can suffer from too many cooks in the kitchen, resulting in a lack of clear direction.
• Conflicts of interest can lead to delayed decision making, which hampers productivity.

Solution

• Each Scrum team should have a single PO. 
• Multiple stakeholders are fine and often encouraged, but one person needs to be empowered as the decision-maker.

Anti-pattern No. 3: PO is not available or doesn’t have time

A lack of availability with product owners is often caused by other anti-patterns, leading to inefficient and counterproductive behaviors. 

Problem

• An unavailable PO creates a slow feedback loop, which causes a reduction in efficiency, capability and morale.
• In prolonged cases, this can cripple the product and de-motivate a team.

Solution 

• PO’s can’t attend every minute at work — but they should be available on-call if needed.
• Always make time to attend the team’s sprint retrospective. They are the single most important ceremony for improving quality. A ceremony is a type of Scrum or Agile meeting. A retrospective is a meeting where the team reflects on how they’re doing and find ways to improve. 

Anti-pattern No. 4: The proxy PO

When the real PO isn’t available, the responsibility usually falls to a department or team lead — or even worse, it defaults to the most senior person. This highlights a poor understanding of the role and a dangerous assumption that the most senior person is the most qualified.

Problem 

• Proxy PO assignments undermine the original concept of the PO role by removing the need for consultation before decision making.
• This leads to delayed decision-making with slow feedback, causing conflicts with direction and prioritization.
• Distrust and disengagement from the team follows, which is almost impossible to recover from.

Solution

• Choose one and stick with it: Either engage the real PO or fully empower the proxy.
• If this isn’t possible, consider employing specialists for additional support, like business analysts.
• Use experiments to remove delays from decision making. In this context, running experiments is the process of testing a hypothesis using data (e.g. A/B testing) to answer a question rather than relying on decision-making from an unavailable individual, who might use a combination of experience, knowledge and gut feeling.

The post Anti-patterns of a product owner — and how to avoid them appeared first on Signifyd.

Welcome to the challenge! 

When I joined the Signifyd Belfast engineering as a team lead in April, I started working on one of the teams that  manages Signifyd’s data and satisfies our reporting requirements for internal and external customers. My biggest project so far was to rethink and redesign how we move data from one store to another, and to hone the most efficient way of utilizing it. 

One of the biggest challenges for Signifyd is how we handle and store the ever-growing amount of data that moves through our systems. We now handle millions of transactions daily from thousands of global customers, which provides opportunities for significant insights into patterns and behaviors and challenges in how to best manage this volume efficiently and intelligently.

The Initial Setup 

Over the past few years, we’ve revisited the technology stack used throughout our system, including our data storage technologies and transportation between them. Like many software companies, we have leveraged a myriad of tools and storage mechanisms to meet our internal and external needs. This introduces issues with maintenance and updates to varying systems, as well as potential inconsistencies appearing in the data available or the “freshness” in replicated stores. As Signifyd matures, so has a lot of the infrastructure and toolsets available, especially in cloud computing and data management. 

In one of the earliest examples of transporting data from primary to secondary stores, we moved our core transactional data from a Cassandra data storage and replicated it to Redshift instances within our Amazon Web Services (AWS) cloud-based infrastructure for analytical and reporting purposes. The initial implementation used a mix of AWS specific technologies to quickly spin up an Extract-Load-Transformation (ETL) pipeline. 

There were a few missteps along the way. We tried writing record IDs to files for querying (not the most efficient approach!), then later settled on a solution of using Kinesis Firehose to consume events triggered by data changes and track these for transfer later. By storing the specific references to the records updated, we were able to move off the previous approach. This approach involved running queries using a component called Solr which had to search across all records for those updated or created within a given time period. As our database got bigger this query took longer to run and return values. 

Once we did that, we used AWS Data Pipeline by manually coupling several pipelines to perform the Extract, Transformation and Loading elements required for a given period on a recurring schedule, like new or update transactions over the course of a 15 minute window.

This diagram illustrates our Data Pipeline Process 

Time to rethink

This approach worked well enough to allow the data to flow through, but over time, and as Signifyd’s customer base grew, the amount of records that were being created or updated over the 15 minute window increased drastically. The upturn in data processed caused the three stages to take longer and longer to complete, to the point at which a single execution of the ETL took more than 15 minutes causing increased delays in data delivery. As a consequence of this we split out these pipelines further to throttle how often certain parts of data were copied across. But all this workaround did was to cause further unwanted lag, increase the need for maintenance and create a growing problem as we became more successful. 

Our initial solution talked directly to our primary data store to obtain the latest record state. This data store is a NoSql database called Apache Cassandra, which allows for the fast retrieval and writing of data, and can handle vast quantities of records in a reliable fashion. However by our ETLs transferring ever-increasing volumes of records in each 15-minute window, we also increased the amount of additional requests being made against Cassandra to pull out the current record state for each. As Cassandra is a mission critical part of the Signifyd infrastructure, and leveraged by other systems which require it for more important operations, the increased traffic caused unnecessary performance impacts and used up resources that could otherwise be used elsewhere. The increased data transfer flow was also prone to failure, due to the delicate binding together of the pieces involved. It caused major issues like Firehose writing data for updates, or activities in the pipelines failing at various points with minimal logging or error tracing. 

Recovering such failures grew more cumbersome over time, and more susceptibilities appeared. The nature of the setup made building a new AMI (Amazon Machine Image — essentially the information needed to create a new instance of a virtual server) to resolve such problems more painful for the team.

Plus, the original approach only served a single data store, Redshift. As our data engineers sought to interrogate our data further for patterns and signs of risk, they only had two options: 

• Use the current Redshift clusters setup, which was already under a significant load and also not the most efficient tool for their purposes
• Write their own pipeline to pull data from Cassandra to another store

My team chose the latter, where we created a new additional ETL which used a different technology to load and write the data out. The data was written out in formatted files into S3 (Simple Storage Service), which is essentially a file storage solution offered and managed by AWS. It caters for large objects or files to be stored and retrieved quickly, with security and reliability built in. The data in these files were then queried and accessed by a combination of
Apache Spark (more on that later!) and Hive Tables, which is an open source data warehouse system to help analyze large datasets.

The way forward

The inconsistencies in the approaches above, how data was stored and the effort replication, led to a concerted effort to design and develop an infrastructure that was more efficient, resilient and standardized. It crucially also allowed other use cases or teams to access the data without requiring a rework or reimplementation. By reusing some of our existing components and building on top of them, we were able to come up with a solution that catered to our needs. 

The first approach was to define a standard version of our underlying data model that could be interpreted by all our datastores to help unify the team’s understanding and reduce potential inconsistencies. Then, we split the process for how we transferred data from our primary store to other stores, and added a single Spark application to consume and use the events raised from data changes to write the underlying records to an offline store in S3 using Spark and Hudi for faster updates. These records are stored in the previously defined canonical format. As we write these updates to S3 we maintain a ledger of changes for other applications to use later when consuming.

Once in this offline store (sometimes called a Data Lake), the data can be accessed directly over S3. For the initial implementation of transferring this data to Redshift, another Spark application inspects the ledger and pulls a number of updates to be written in batches. This helps throttle the data writing during peak periods. The application uses the same underlying format for interpreting and writing the data to a new Redshift schema which matches this offline structure. It also inspects the current schema as persisted in Redshift and maps to the canonical format to reduce any code changes as new or existing record definitions alter. With the data read and interpreted, it is then loaded into staging tables within Redshift for temporary storage. For example: for customer information we might store it in a staging table called customers_staging. This table would mirror the destination tables structure but contain only the records for the current execution. When the data is in the staging table it is then written to the target table, for example customers, by performing either an update to an existing record with the same ID, or insert a new record (this is also known as up-serting).

This diagram illustrates our Spark ETL Process

As part of the above process, we replaced a large amount of the ETL processing with Spark applications running in Amazon EMR (Elastic MapReduce)  – this is a cloud based platform built to process vast amounts of data quickly and cost-effectively. We leveraged Spark for this process for a number of reasons:

• Speed of processing data: The engine is around 100 times faster in memory during execution, by reducing disk read/writes and leveraging in-memory computation.
• Ease of use: Spark comes with a rich set of APIs and operators for transforming and working on data, especially in large amounts.  
• A unified engine: Spark supports a large number of operations, from SQL queries, data streaming, machine learning and graph processing. We can boost productivity by combining complex workflows.
• Multiple language support: We can use key languages including Scala, R, Java and Python with this tool.
• Community spirit: We’re part of the expanding Spark community, a group that contributes to the framework’s underlying source code to add features or fix bugs, as well as help develop related frameworks and toolkits.

Future work

Completing this initial piece has allowed the Signifyd engineering teams to plan for enhancing our reporting. One area for future development is transitioning our reporting using Redshift as the primary offline data store. We currently maintain a full history of data with Redshift, which is largely unused for records older than a couple of years — a detriment that increases data size and cost and can reduce query performance. 

Our enhancements provide a standard format for our records and allow us to use other technologies such as Redshift Spectrum to expose certain data from our S3 store rather than explicitly Redshift. This will reduce database size (and costs) and improve performance for most of our use cases which focus on more recent data. 

It also allows our teams to maintain entire history access for our internal and external customers. We are also looking at running these Spark applications within our own Kubernetes clusters rather than Amazon EMR. This will make testing and development much easier, and reduce dependency on another component for execution.

The ability to work on emerging and modern technology stacks, and focus on solving real problems for our customers, makes my job a challenging and rewarding learning experience every day. Technology and our customers never stand still, so neither can we!

The post Technology never stands still appeared first on Signifyd.

What I think of as my first, first day came after my husband and I moved to Silicon Valley from the east coast. Dan had just landed a new job with a well-known valley company. I was looking for a job. And I was pregnant — midway through my pregnancy, in fact.

I was pregnant and Hispanic and a woman in software engineering — a minority on all three counts. No matter. After evaluating my qualifications, Signifyd offered me the position for which I had looked and prepared.

 Agile scrum meetings, flat-structure code reviews and foosball

This position became one of my favorite jobs. The atmosphere was professional, friendly, and productive. We held agile scrum meetings that started and ended on time, while keeping on purpose. We had flat-structure code reviews. Management encouraged us to share our ideas for improvement with any management tier.

Sometimes we played games like darts and foosball. Someone was always available to help overcome roadblocks. Retrospective action items were implemented; team members ironed out issues; our manager maintained an open line of communication. The technical work offered learning opportunities.

In the autumn of my first year at Signifyd, our son, Benjamin, was born. Dan and I experienced all the joy and exhaustion of new parents. And, of course, the day eventually came when it was time for me to return to work.

I read a Recode story recently about mothers returning to jobs in tech. It talked about some of the struggles for some moms and some of the great things some companies are doing to make things easier for those returning to the office. The story included a survey that found that on a scale of one to five (five being “great”), 48.5 percent of returning mothers rated their experience coming back to work as a four or five.

More than a quarter ranked their returns as a one or two, with one being “horrible.”

I am squarely in the first group.

After the birth of our son, I asked Signifyd for some help to better adjust to my new life as a mother. The company leaned in. It provided flexibility with my work hours. And it allowed me to work remotely for a while to balance motherhood and work, an arrangement I’m pleased to say that 62.5 percent of the mother’s surveyed by Recode were able to experience. I do feel for the other 37.5 percent.

 In the midst of rapid change, culture can remain constant

Eventually, I took a leave to care for my family full-time. I kept my hand in the game during that time, working on a gaming app that’s coming along nicely.

By January of this year, the time was right to rejoin Signifyd. I love my work and being a woman in tech. Programming is something I’ve wanted to do since I was a kid in Cuba, playing video games and taking programming classes using Logo to animate a digital turtle.

Signifyd hasn’t changed much since my first, first day. I’m working with a new team, but I still like my job — for the same reasons I liked it before.

It’s fulfilling to be working on innovations that are helping ecommerce businesses to fare better in the face of fraud. I like to think that the work we are doing here is helping others do the work that they always wanted to do.

Meanwhile, I hope that Signifyd continues to adhere to its start-up values as it grows. It remains fresh with innovation and professionalism, which, in my world of software engineering, often manifest themselves as interesting work, friendly collaboration, focus on quality, and caring for employees and especially women in technology.

Photo by William Iven on Unsplash

The post A mother and software engineer finds you can design a balance in the world of tech appeared first on Signifyd.

Early on this year, I saw an online job posting for Signifyd and after doing a bit of background checking, the company seemed like a great match for me. I had to weigh up continuing my Ph.D in a field that I am passionate about, or going into a slightly different industry to gain more backend development experience. Signifyd offered that chance, with a really interesting technology stack.

The company, based in Silicon Valley, California, had just set up an R&D hub in Belfast and it sounded like a good opportunity to join a team that was being built from the ground up. Even better, the role involved machine learning and that fit well with my area of interest, so at the time I didn’t feel like I would be leaving anything behind.

The interview process was quite intense, but wasn’t a drawn-out process, and involved a coding challenge followed by a whiteboard exercise where I had to talk through a piece of software I had developed and answer questions about my product. The coding challenge stood out to me. Signifyd had put a nice spin on it from my past experiences: First they asked me to understand the code and then discuss how I would suggest improvements, similar to a code review. Once I had finished the interviews, the recruitment team was quickly in touch with an offer. That’s when I had to make the big decision: stay in research, or join Signifyd. I am sure you can figure out which choice I made.

When I start a new position anywhere, there are a few things that are important to me:

• How prepared is the company and the team for a new person joining?
• Is everything set up and ready for me?
• How quickly can they support me to get up to speed?
• How comfortable is it to ask questions if you need to?
• Will I feel useful early on and feel like I’m contributing?

I have to say, Signifyd was great in every regard. In a previous role with another IT company, I felt thrown in at the deep end and left to figure things out myself; painfully. Thankfully this was not a repeat experience with Signifyd. Now a few months in, I feel part of the furniture and I’m helping newer team members get up to speed and settled in.

It also helped that before joining Signifyd I was invited for lunch with the Belfast team the week before. This was a valuable experience. I got to meet the team in an informal environment to get rid of some of my pre-first-day nerves. Talking to the team at Signifyd over lunch made me excited to start the following week.

On my first day I was introduced to everyone within the office and the team I would be working with. They walked me through the different apps Signifyd uses on a day-to-day basis: Slack for communication, Zoom for connecting to meetings and Jira for agile project management. We use IntelliJ code environment and Phabricator for our review tool. I had worked with IntelliJ before briefly, but not with Phabricator. Dean, my assigned “buddy” for my first week, showed me some useful features and commands, which helped with the learning curve.

With Dean, I initially went through the process of getting my Macbook up to date with all of the tools and software required to start developing before cloning the codebase, building it and importing into IntelliJ to prepare for the next day.

On day two they gave me a small user story to get me going. This enabled me to make the required changes in code and push to production as quickly as possible. This was a great feeling on my second day, as I was already starting to become familiar with the environment I would be working with!

Over the next day they assigned me a whiteboard session that gave me a general overview of the current software system. They also explained team goals to me. I thought this was a fantastic chance for me to ask questions and it provided me with a more holistic view of Signifyd.

To finish off my first week, my team continually worked to provide me with user stories that would best suit me as the opportunity to learn the ropes quickly. Throughout the week I got to experience the relaxed and friendly working environment, ranging from going for lunch together everyday (provided for free!) to participating in daily stand-up meetings. This made me feel like part of the team after only my first week. I am well into building Java microservices for our product after a number of weeks into my role!

Overall, my on-boarding, and the support I had, made it possible for me to move my work to production so quickly. This has allowed me to feel productive and to feel that I’m adding value for the people I work with.

It’s been a truly positive experience. Our office environment is fun, relaxed and friendly, and you get to meet everyone including the people outside engineering. It’s how I imagined a real startup to feel, with new people joining on a weekly basis.

If you’re wondering, the big question did pop into my head: “Did I make the right decision to move?” I’m pleased to say, it was an easy answer: Yes! It’s a great environment, my work is fun and I am getting to learn new things daily. I’ve already referred my friend Chris who’ll soon be joining our growing team.

I know he’ll have a great first week like me, but my code will always be better!

Laptop photo courtesy of Unsplash; Robert Finn photo courtesy of Signifyd

The post Pushing code to production on day two: My experience joining Signifyd appeared first on Signifyd.

In the race to find the best, most qualified candidates, selecting the best one from a pool of applicants is not a solved problem. Conversations tend toward one of three topics:

1. Espousing the virtues of one technique over another
2. Focusing on the impossibility of predicting productivity from a bit of coding performance art
3. Declaring, “This is how it’s done at my company,” because we’re pretty proud of how we do things

This article is about what we do when hiring for U.S.-based engineering roles at Signifyd: #3.

I suggest everyone incorporate a Code Review equivalent into their own process.

The theory behind our process

First a short digression into theoryland: It’s important to understand Signifyd’s overall approach to interviewing. For some context, read up on the Unified Arguments for Overall Approaches. You might recognize that as a hedgehog way of thinking.

Conversely, the fox way of thinking relies on a large toolbox of diverse approaches, even if the combination is theoretically unsound. Our hedgehog approach to designing an interview scheme is based on a fox approach to interviews.

I will explain how we’ve found success with this approach, as well as the downsides, plus future improvements. The final piece of our engineering interview process is my favorite aspect of our process: the code review. Of all the steps in our interview process, this is the step I enjoy the most. It’s also the step I have heard the most positive feedback about. In fact, I’d recommend a code-review aspect for everyone’s interviewing process.

Random forests, foxes, and amateurs

Meaningful interviewing is hard. Maybe impossible. Most people aren’t good at it, and many companies struggle with it. Folks can become consistent at executing some hiring processes, but despite their best efforts, they let poor candidates through to the hiring stage, or they exclude candidates who would have been a great fit for the job.

Google, for instance, has a very specific type of filter. Engineering candidates must be very good at “standard whiteboard algorithm problems” to make it through their gauntlet. Companies like Google can get away with a very strong and specific filter because their candidate pools are so large. Most other companies can’t do this.

Without a very strong and specific filter, what can we use to hire well?

The core of Signifyd’s business—the unique value we create which we then convert into value for merchants across the world—is predicting whether or not a given transaction is fraudulent. Sometimes merchants’ fraud traffic patterns are similar to each other, but we often onboard merchants with previously unseen types of fraud. The machine learning models we’ve chosen to tackle this problem are random forests.

We build these random forests by first building one illogical decision tree. Then we build another, and another, and so on, and average their results. This is like leveraging the wisdom of the crowd, which works like a charm. We apply this approach when interviewing Signifyd engineering candidates.

We know the following things are true: We’re amateurs. Interviews give a weak signal about future productivity. We don’t have a Grand Unified Theory of how to get the most out of an interview. Instead, our Grand Unified Theory of how to get the most out of an ensemble of interviews is based on diversifying, gathering as many uncorrelated signals as we can, and mushing those weak signals together.

A quintessential fox approach.

Our weak predictors

Here’s a quick look at what we do in practice and what weak signals of ability can look like:

• A timeboxed (for real!) take-home, about 50 percent design and 50 percent algorithm
• A “standard whiteboard algorithm” problem geared toward “can you code?” and introspection over difficulty or cleverness
• A code review, where candidates annotate, eviscerate, and/or rewrite some provided problematic code
• A technical deep dive, where we delve into interesting bits of the candidate’s technical background
• A design problem, where the focus is on good code structure rather than algorithms
• A common systems design problem, like design Bitly

Not every candidate gets every category from the above list. Occasionally one category is doubled up in the interview process, based on the candidate’s experience and background and the position they are interviewing for. But generally, our candidates get at least four of these six.

Pros and cons:

Code reviews

Have I mentioned that the code review is my favorite part of the process? It seems to be well regarded by our candidates, too. Our original code review question contained a bit of real production code that was simply bad.

It was very hard to read, so we started the code review with a request: “Please read this undocumented function and then describe what it does in your own words.” Then we asked candidates to write good documentation, assuming the code was 100 percent functional. We asked them how they’d refactor the code to be maintainable, and finally, we asked them to rewrite the function.

Now we have a few types of code review questions. Some feature a difficult-to-follow, small function as above. Some feature a larger, poorly designed component. In this case, we would ask the candidate to discuss the design and implementation with us as if we, the interviewers, were junior coworkers who just submitted the code for review.

This type of question is an excellent test. It’s more in line with the practical skills we use in our day-to-day work when reading legacy code, reviewing peers’ code, writing empathetic comments (where we try to include encouraging emojis :D), and rewriting bits of code to optimize for readability and maintainability. This is far more common for our teams than needing to solve a tricky algorithm problem. It also gives a clearer picture of how the candidate views code reviews, rather than just asking, “Do you do code reviews?”

Current results and future plans

Theory is great, but you’re probably asking yourself, “How well does this process work?” In the interview process, you can’t know how well your declined candidates would have done in the role and you won’t know how good your hires are in their roles until a few months down the line, at least.

What we do know is that is that we’ve been able to maintain rapid growth, while assembling a talented team of engineers, the members of which appear happy to be here as they tend to stick around.

At Signifyd, we’re still working on our process for effective software engineering candidate interviews. We’re working to add more variety in the interview questions we ask within each category —not just because it’s good practice to routinely cycle out old questions, but because interviewers can get tunnel vision when they ask the same question too many times. We should strive to be well-calibrated on the questions, but there’s a fine line between well-calibrated and stuck in a rut.

We’re still exploring best practices for interviews for our technical and engineering positions in Belfast, as the culture is different and candidates expect different things. Similarly, we are expanding our repertoire of weak predictors for data scientists. Overall we like our process and plan to iterate on it, not revamp it, in the foreseeable future.

All of this is to say, we’re hiring! Now that you know a little more about our process, check out our careers page for open positions.

photo courtesy of UnSplash

The post How we interview for U.S. positions: A prep guide for engineers appeared first on Signifyd.

It was also a typical remote experience — I was the one remote person on the team. My managers had to go out of their way to include me in discussions and I flew down for a week each month to stay on top of things.

When I joined Signifyd three years ago, a major attraction was that most of the engineering team worked remotely. Since then we have more than tripled the size of the team and are now split more evenly between remote and onsite engineers. Along the way we have refined and improved our social norms, our ways of interacting and our organizational structure to minimize the offsite/onsite distinction and maintain strong team cohesion.

One tool that’s proven surprisingly effective is Sococo, an app that gives us virtual office space and tools for communication. This screenshot shows a random day in the office:

With a glance, I can see in the top right that the ADE team is meeting, with one member sharing her screen.  I can tell that a couple of teammates are having a video chat in the Falcon Heavy room and see that one person is away from his desk (Lunch, bottom left). With a click I could join one of the ongoing meetings, jump into the Ping Pong room to chat with the coworker there or move to the CoffeeBreak room to signal a short period away from my keyboard. Ordinarily, I’d also see my coworker’s names, but we’ve removed them from the illustration for privacy reasons.

When a virtual office feels like the real thing

It’s funny how much this feels like an office to many of us. If two people step out of a group room to continue a conversation, they’ll naturally step across the hall.  When we had to move to a bigger virtual space, debates about layout were intense and there was a scramble (and some hurt feelings) when each team wanted to claim the best rooms — those with comfortable couches and decor. And we’re engineers!

It’s easy to get remote workers on board with a technology like this. The bigger hurdle is to make it work for those onsite.  For example, they are asked to negotiate two work environments and choosing to be fully present with headphones on in Sococo sometimes doesn’t make sense, because it requires being less fully present in the physical office.

We’ve connected a physical room in San Jose directly to Sococo and played around with mics and headphones to improve audio in noisy settings.  We encourage non-engineers to become fluent in the tool. My team’s product owner is a pro – if he has a question, he’ll pop into our team room turn on his camera and mic and speak up. Invariably one of us will engage and the rest can listen and chime in as necessary.

Despite the benefits and fun, Sococo is not a replacement for physical proximity and we constantly face the challenges of building successful teams across large distances. Onboarding for both onsite and remote engineers is more difficult. Managers need to handle the unique challenges faced by remote teams. Individuals need to become comfortable reaching out to a remote colleague as quickly and informally as they would to someone sitting at a desk beside them. Communication between individuals, teams and departments is a common topic in agile retrospectives; the ability to stay focused without losing context isn’t unique to remote work, but it is even more critical.

Those of us who work remotely really appreciate the effort that those throughout the company make to include off-site engineers in company-wide meetings and events. We’re incorporated into all-hands meetings, lunch-and-learn sessions and monthly cross-team lunch meetups. In addition to Sococo, the company has invested in hardware so that most conference rooms can be used with Zoom teleconferencing.

But some things are simply impossible when you work remotely. So, the entire engineering team gathers a few times a year at headquarters for a fun onsite week where we participate in workshops, do lots of whiteboarding and, most importantly, hang out and get to know each other.

Remote work recognizes real life

And, of course, there are advantages to working remotely.  As long as remote engineers are available during the appropriate hours and getting their work done, an engineer could stay home to let in a plumber, skip a bad Bay Area commute to hunker down and focus on a big deadline or fly out of state to support a family member with a health concern. Remote engineers know they can work successfully from afar; no changes to team communication or daily rituals required. In fact, it’s great for everyone at the company to experience remote work at some point.

Our remote culture has been a work in progress since the company’s inception, with most of the original engineers, as well as several managers, working remotely. Heck, one of the founders still works from three time zones away! There’s no concern that the company will suddenly abandon this journey. We expect to continue to tackle unexpected challenges that arise, expand the suite of tools we use to communicate and evolve our teams to meet the onsite and offsite challenges of a growing company.

The post Far apart, but working closely together appeared first on Signifyd.

From the pitch email:

“They actually have coding standards and do real code reviews. They pay for the tools you want and will honestly deprecate tools that have outlived their usefulness.”

I went on to talk about engineering rigor, the tech stack and some cool projects, but that was my lede. I chose to focus on those things because I felt they exemplified Signifyd as an engineering culture. In my moderately long and modestly successful career, I’d never found that kind of disciplined approach to making good choices for now and later. I thought it was inspiring.

Months later, I found an entry on our internal wiki entitled “Do It Right,” written by one of our co-founders. At 345 words, it’s a short entry. It’s a little dry and prescriptive and it’s very much geared toward writing code. However, it is the living soul of our culture. In it, Mike Liberty discusses the tension between creating clean, elegant code and delivering value to the company. He then goes on to set forth rational rubrics for choosing when to put in a little extra effort right now and when to do the expedient thing now and come back to the issue later.

Chose to do it right

The first part of the rubric is very simple: If doing something in a better way doesn’t balloon the scope of your task unreasonably and won’t cause you to miss a commitment, then do it the better way. This isn’t a suggestion. It’s a directive. At Signifyd, we believe that it is our duty to provide better code — for our customers, for ourselves, and for our colleagues. So when given the option, we choose to do it right.

The second half of the rubric is murkier but still important. If for reasons of scope or delivery you can’t do something in what you feel would be the right way, then you are responsible for making sure we know to revisit the issue. Again, we take this more as a responsibility than an option. While we acknowledge the reality that sometimes we’ll have to choose shipping over creating the most polished solution, we don’t believe in letting that compromised decision linger longer than it needs to.

A little over a year after I’d pitched my friend on Signifyd, he and I were working on a team together. We had a project looming that would help us handle the influx of Black Friday traffic and I was keeping my eye on the calendar. In the midst of all this, Mike Liberty’s code reviews made me realize that some of the code I was creating wasn’t in line with our Do-It-Right culture. And even though I felt some pressure from that immutable deadline, I reworked my code until it was done well. In the end, I was happier with what I’d created.

Ironically, in the scope of those code reviews, I discovered that as a team, Signifyd was great at making sure code was thoroughly reviewed, but not so great at making sure the author of the code felt great about their contributions. Because code reviews were so intrinsic to doing it right, I agreed to rewrite our code review guidelines with an eye toward empathy, respect, and clarity, as well as toward code correctness.

A new way to review code

After the new guidelines had been vetted by a group of senior engineers, I presented a hands-on code review workshop. Over those two hours, my colleagues asked incisive questions and raised issues I’d never thought to handle. Unlike such training I’d given elsewhere, they soaked it in and really thought what great code reviews could look like. It was inspiring at the time, and several times since, someone has told me how much impact my workshop had on their work style.

The Do-It-Right culture was so ingrained in our work that its central message was codified as one of Signifyd’s six core values: Design for Scale. The value is further described: Build solutions to scale and last. We are here to stay.”

One of the core aspects of this culture is that we’re never done growing or innovating. In much the same way that we very carefully updated how we review each other’s code, we carefully weighed our options when choosing GRPC as our microservice framework. Once Slack had surpassed our incumbent chat service, we made a plan and migrated. When we saw our colleagues struggle with onboarding, we created tools to get people up and running quickly — and every new hire has kept those tools polished.

Those few examples are just the tip of the iceberg. We’re often presented with moments where we are able to choose to do it right. Those moments are big and small, but we do our best to treat them with appropriate deliberation and care. Our culture not only allows for that level of craftsmanship but encourages it.

Whenever we get a chance, we all do our best to make Signifyd a better place to work. I can’t pretend everything goes as planned, but I can say that not a week goes by when I’m not amazed at how willing we are to grow as a team, how willing we are to do it right.

The post Signifyd’s do it right culture appeared first on Signifyd.