Troubleshooting

See the following sections for common problems and ways to solve them.

Run doctor

Run devtools doctor to determine if your environment is set to run DevTools.

Ensure the environment is setup correctly

It can be useful to ensure everything is in a clean state. The following should ensure that

  • devtools stop stops the docker machine and cleans up networking
  • eval "$(devtools config)" clears environmental variables docker uses to communicate with the docker host
  • devtools start starts the docker machine
  • eval "$(devtools config)" sets environmental variables docker uses to communicate with the docker host

Ensure your images are up to date

From time to time images are updated to fix bugs or add functionality. You won't automatically receive these updates but you can fetch them when you hear new ones are available.

docker pull imagename can be used if you want to update a specific image. For example, if you wanted to make sure you had the latest dnsdock you'd run docker pull phase2/dnsdock.

docker-compose pull can be used within a project directory to make sure you've got the latest version of all images in the docker-compose.yml file.

Configure Your Shell

If you do not have any containers listed when running docker ps or you get an error message like:

Get http:///var/run/docker.sock/v1.20/containers/json: dial unix /var/run/docker.sock: no such file or directory.

* Are you trying to connect to a TLS-enabled daemon without TLS?

* Is your docker daemon up and running?

Or an error message like:

Couldn't connect to Docker daemon - you might need to run `boot2docker up`.

Make sure your shell has the necessary environment variables by running:

eval "$(devtools config)"

Reset everything

If a problem continue to persists and the Docker Host is non-responsive, you may need to resort to the nuclear option of blowing everything away and starting over. Note this is a nuclear option as even your persistent data area will be removed if you don't back it up. If you have any data that needs to be maintained be sure to get a copy of it off of your VM first with the scripts provided.

To wipe everything out and start over

  1. First backup your existing data (if desired) by running devtools data-backup. This will sync your entire /data directory to your host machine.
  2. Then you can run devtools remove This removes the broken Docker Host and it’s state, making way for a clean start.
  3. Next you can rebuild everything by running devtools start. This will create you new Docker Host.
  4. If you wish to restore the /data directory that you previously backed up, then run devtools data-restore

Files Not Found

If you get messages about files not being found in the container that should be shared from your host computer, check the following:

  1. Is the project checked out under the /Users folder? Only files under the /Users folder are shared into the Docker Host (and thus containers) by default.
  2. Does the docker-compose.yml file have a volume mount set up that contains the missing files?
  3. Get shell in the Docker container via docker exec -it <container name> /bin/bash and check if the files are shared in the wrong place.
  4. Get shell in the Docker Machine with docker-machine ssh dev and check if you find the files under /Users.

Image not found

If you encounter the following error about the Docker image not being found when starting a project, it may indicate that the image is private and your Docker client is not logged into a required private Docker Hub repository:

Pulling repository phase2/privateimage

Error: image phase2/privateimage:latest not found

To solve this, run docker login and provide the relevant credentials.

Docker client and server version incompatibilities

If you see an error similar to this:

Error response from daemon: client is newer than server (client API version: 1.20, server API version: 1.19)

You likely need to upgrade your Docker Host to a get Docker client API compatibility. Do that with:

`devtools upgrade`

Network timed out and can't pull container image

Example:

Pulling cache (phase2/memcache:latest)...
Pulling repository docker.io/phase2/memcache
Network timed out while trying to connect to https://index.docker.io/v1/repositories/phase2/memcache/images. You may want to check your internet connection or if you are behind a proxy.

Try restarting docker-machine: devtools restart

Started machines may have a new IP address

Example:

Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
[WARN] Docker daemon not running! Trying again in 3 seconds.  Try 1 of 10.
...
[WARN] Docker daemon not running! Trying again in 3 seconds.  Try 10 of 10.
[ERROR] Docker daemon failed to start!

The Docker Host (probably called dev) has it's IP changed. The generated TLS Certs are no longer valid and must be regenerated.

Possible causes or relations/patterns:

  • Another VM was started in VirtualBox.
  • Machine went to sleep and somehow caused issues with the running VM.

Fix:

Try running: docker-machine env

If that does not work, you should be able to start the VM directly through docker-machine: docker-machine start dev

Next, check the IP / TLS status by running: docker-machine ls

The output will likely report something akin to:

dev       -        virtualbox   Running   tcp://192.168.99.100:2376 Unknown
Unable to query docker version: Get https://192.168.99.100:2376/v1.15/version: x509: certificate is valid for 192.168.99.101, not 192.168.99.100

Now, regenerate the TLS Certs: docker-machine regenerate-certs dev -f Devtools should now be able to start. Don't forget to run `eval "$(devtools config)" after.

Containers Started but Service Not Available

Your Docker Host is running, the project's containers are up, and command-line operations work fine. Why can't you view the site in your web browser?

DNS Services

The DNS services that DevTools spins up may not be working. Run docker ps and see that you have a dnsdock and dnsmasq container.

If those services are not running, try devtools dns to bring them up, or a full devtools restart if that does not work.

devtools dns-records is also useful to see what containers have registered names.

DNS Configuration

It is also possible that the DNS services are running for your environment, but somehow the configuration is wrong. Run devtools dns-records and make sure your project has an entry. If not, you may need another restart, or perhaps you are missing DNSDOCK_NAME and DNSDOCK_IMAGE environment variable entries in your docker-compose.yml manifest.

Slow-starting Services

Some services, such as Apache Solr or Varnish, can take longer to start up than Apache and PHP-FPM. As a result you might load the browser so fast that not all services are available, which in the case of a proxy may prevent the page from loading at all. Wait a short time and try reloading the page.

Failed Health Checks

Some services such as Varnish depend on others to operate, and have built-in health checks to verify the other service is operating.

If such a health check fails, there could be two problems:

  1. The internal DNS routing between Docker containers is broken. Make sure the configuration of your services is correct.
  2. The dependency (e.g., Apache behing Varnish) is not yet up and running when Varnish performs its checks.

In either case, you can often repair the problem by performing a clean restart of the broken service.

docker-compose stop proxy
docker-compose rm -f proxy
docker-compose up -d proxy

Your other services should already be up and functional when this is done, so the health check will not fail on account of (2).

Checking on Varnish Health

If you suspect Varnish may be failing, run docker exec -it [VARNISH_CONTAINER] varnishlog and scan for VCL compilation errors.

Service Became Non-Responsive

Sometimes a service locks up. Apache stops serving results, Solr stops indexing or responding to search queries. These things happen on servers all the time. It may even happen more often on Docker, especially since we are now using more "infrastructure" in our local environments.

Docker is meant to easily sandbox these problems from the rest of your machine, and to easily resolve these problems by allowing you to dump the problem and start over fresh very easily.

Hard reset on a service (as describe above), is very much like a power cycle:

docker-compose stop [BROKEN_SERVICE]
docker-compose rm -f [BROKEN_SERVICE]
docker-compose up -d [BROKEN_SERVICE]

Sometimes data for a service is volume mounted from outside the container. This persistence is good when the data is healthy, but can be really bad if the data is part of the problem (e.g., broken lockfiles).

docker-machine ssh [MACHINE_NAME]
rm -Rf /data/[PROJECT]/[DIRECTORY_FOR_SERVICE]

Remember that your service may have configuration or data in another service, you may need to perform this operation against multilpe containers (e.g., Solr)

NFS Conflicts with other Environments

Running the Devtools VM in conjunction with other development environments (such as Vagrant) can sometimes cause conflicts with volume mounts. This is due to the fact that the Devtools VM mounts the entirety of /Users into the VM on OS X. Additional NFS mounts from other environments that target subdirectories of /Users will fail.

The easiest workaround is to keep projects that use non-Devtools environments like Vagrant in a directory outside of /Users, such as /opt. Alternatively, you can run a single VM at a time and manually clear out /etc/exports prior to switching environments/projects.

Migrating Vagrant boxes outside of /Users:

If you'd like to move your vagrant boxes outside of your home directory, perform the following steps:

  1. Choose a destination folder. We'll be using /opt/vms for this example. Insure the destination has enough free space. Typically this is not a problem because our Macs are one single partition.
  2. Make the new directory, and ensure your userid owns it: sudo mkdir /opt/vms; sudo chown -R userid:userid /opt/vms
  3. Add export VAGRANT_HOME=/opt/vms to ~/.bash_profile
  4. Move your vagrant folders over to /opt/vms.
  5. edit /etc/exports, updating any vagrant mount point to use the new location. (example: /Users/userid/vagrant/projectx/cms)
  6. While a reboot isn't neccesary, it'll help to make sure nothing is running and using the old mount points.
  7. cd into your new vagrant directory, and do a vagrant up.