Internal process of deployment in Cloud Foundry

As of now we have learnt what is Pivotal Cloud Foundry and how to create application and push it into Pivotal Cloud Foundry. Now, we will look at what is the internal mechanism of an application deployment in Pivotal Cloud Foundry or we can also say that what is the cf push sequence mechanism.
As we already know, we need to use cf push command from CF CLI to push/deploy application into Pivotal Cloud Foundry. But most of them doesn’t know the internal process of pushing an application into Pivotal. Take a look at below sequence diagram to understand more about the deployment process of an application in Pivotal.

As shown in the image, there are few important stages which we will come as part of the application push is,

CLI

Cloud Controller

Blob Store

CC_DB

Cell (Staging/Running)

As shown in the above diagram, 12 steps are running internally to deploy an application into Pivotal Cloud Foundry. From step 1 to step 7 is part of Staging and 1 to 9 steps will part of Droplet of the application. Diego will be the responsible for running Droplet. Will look at each and individual steps which are show in above diagram,

1. As shown in the image CF PUSH is the initial command which we need to issue from CLI to start to push application into Pivotal Cloud Foundry.
2. Cloud Controller is the responsible for taking all the requests coming from CLI and similarly CF PUSH will come to Cloud Controller and create application which we are trying to push.
3. Cloud Controller will send application METADATA to store into CC_DB (Internal Cloud Controller Database). METADATA is nothing but, application name, memory, buildpack and high level information.
4. Application source files and other files related to pushing application jar/war will send to Cloud Controller.
5. The RAW source code and application files are stores in the Cloud Controller database in the format of BLOB or Binary in Blob Store.
6. Having all these application information in  place, Cloud Controller initiates the application startup process inside the container. So application start command is invoked and all the application startup events are triggered.
7. With the help of Auctioner, try to identify which is the available CELL to stage the application. This CELL is also called as Staging Cell. As on when we issue the cf stage command, still this step will be executed internally.
8. Streaming out put of the application staging will be send back to Command Line Interface (CLI) either it is success or any issues.
9. After streaming out the application, Diego creates a TAR image of running application and stores in the format of BLOB. This is also called as droplet. Taking and running Droplet is the responsibility of Diego.
10. After completed application staging, it report back to the Cloud Controller with the output information.
11. Another Auctioner will happen to run the staged application. This auction will get identify the Running cell to run the application.
12. Return the report of the running application back to the CLI (Command Line Interface)

What is the difference between Staging and Restart?
As shown in the image, staging will be in between 2 to 8 steps and 9 to 12 is the restart. Restart will work on top of already staged application

Previous: Application Manifest Usage in Cloud Foundry

Next: Set Environment variables in PCF

Advertisement

Application Manifest usage in Cloud Foundry

In previous post we have seen how to create a sample application and push it into Cloud Foundry and access the application by using Cloud Foundry Route. The current post we will learn about the Manifest.yml usage in cloud foundry applications.
Manifests are ways to describe and deploy application seamlessly to cloud foundry environment. Basically, using Manifest.yml file we can describe how to push our application. In this process, we need to define application name, how much memory that our application required and services that we need to bind and environment variables..etc in manifest file. There is a huge list of alternatives and benefits for any cloud application that uses manifest.

When we try to push our application by using CF PUSH command, it will try to identify manifest.yml file in same directory, that we are trying to push the application. Once it identifies, that process the properties defined and uses to start the application.  If CF PUSH doesn’t find the manifest file, CF push tries to identify what application it’s trying to push. In this process, since it doesn’t find any yml manifest properties,  Cloud Foundry applies default values for all required fields like name, memory & services etc.

We can even provide external Manifest.yml file to CF PUSH by using -F argument. This argument simply tell to Cloud Foundry, go and find wherever it is exists manifest.yml. Once it is located, then it follows the processing as mentioned previously.

If we are going in the way of manifest, there are few mandatory fields which we need to provide,

• application name
• application memory
• buildpack

Buildpack simply indicate cloud foundry about the environment that application should run on. So if we have a java application, then we should provide a java buildpack.

Manifest.yml files are really helpful for CI/CD. Using a standard way with Manifest.yml, we can make our application run anywhere. Even if we changes or migrate the Cloud environment, we have already have configuration in the Manifest.yml that we need to run our application. So, no need to worry about changes after the environment migration.

Manifest.yml files are designed in a way that we can define more than one application inside it. So, assume that we have group of applications in a space. Then defining these applications configuration and pushing into Cloud Foundry is very simple. We can add all manifest properties inside a file and we can push into Cloud Foundry and it is only one time creation and we can use wherever we need. If we need to new service, just we can configure the service in Manifest.yml and redeploy.

Summary:

• Describe Application Deployment
• CF tried to find Manifest.yml in same directory
• External manifest can be provided
• There are required and optional fileds
• Very useful for CI/CD process
• Define more than one application in single Manigest.yml
• Simplest way to deal with CF Application

We can see default properties of application considered by Cloud foundry while starting the app.

name:              cfgreet

requested state:   started

routes:            cfgreet.cfapps.io

last uploaded:     Wed 03 Oct 18:17:54 IST 2018

stack:             cflinuxfs2

buildpacks:        client-certificate-mapper=1.8.0_RELEASE container-security-provider=1.16.0_RELEASE

                   java-buildpack=v4.16-offline-https://github.com/cloudfoundry/java-buildpack.git#41b8ff8

                   java-main java-opts java-security jvmkill-agent=1.16.0_RELEASE open-jdk-…

type:            web

instances:       1/1

memory usage:    1024M

start command:   JAVA_OPTS=”-agentpath:”

How to create manifest.yml?

Cloud Foundry provides the command called create-app-manifest to create manifest.yml file for your application. It will contains all the application information including services which are bind to application. Command syntax look like,

cf create-app-manifest  -p

Below is the sample manifest file look like,

manifest.yml:

applications:
- name: cfcloudgreet
path: target/CFGreet-0.0.1-SNAPSHOT.jar
buildpack: https://github.com/cloudfoundry/java-buildpack.git
memory: 1G

While pushing the application into cloud foundry we need to provide the path of the manifest file. So, we no need to pass all the properties in command line.

Previous: Create Service Instance in PCF

Next: Internal Process of Application Deployment in Cloud Foundry

PCF Application Services

Service is one of the important factor in recommended 12 factor apps also a commonly used feature in cloud application deployment. Services has to be attached to resources either in the local or in production environment. Cloud Foundry build up on that idea where resource and provides the service instances. The basic idea what we have here is,

On-demand resources = Services

We can manage application services in two ways by using CF CLI and another is form marketplace. Marketplace helps you to use readily available services in Pivotal Cloud Foundry. You simply go to services tab in the Cloud Foundry or Marketplace to look services available. Assume that you want to see MySQL database in Cloud Foundry that locates in Marketplace. As on when you bind that service, you have service ready to use. This is also called as Provisioning.

Cloud Foundry services uses a provisions as a service. There is a command for managing services of application. That is, CF SERVICE command. There are two types of service instances,

Manage Service Instance

User provided Service Instance

Manage Service Instance: An instance which is created from the marketplace service is called as Manage service instance. which all services available in market place to bind with our application, these type of services called as managed services. As part of this, we use only create-service and bind-service commands to create and bind service with our application.

User provided Service Instance: An instance which is created from the user provided service is called as User provided service instance. The services which are not present in marketplace and created by the user is called as User provided services. To create user provided service, we use the command called create-user-provided-service. This command also has the alias name called as cups.

List of Service Commands:

Many service instances are advance that we use in Cloud Foundry itself. Such as the MySQL etc. These services are using Service Broker API to implement their all services. Cloud Foundry Marketplace advertise these all services and shows you all available services you can use.

As on when we select a service there will be options shows start, shutdown and bind. These are manage by Service Broker APIs. Once we can get the instance of manage broker API, we can use of it in our applications through binding. We can use CF BIND command in our manifest file.

So, when we pushed our application into Cloud Foundry, that service will be bind to our application. After that we can start using immediately. We can insert data, pull data and so on, it is a database instance.

Will definitely have demonstration of both the sides like managed and user defined services. User defined services are advance that we can provide to our applications and actual user seemly like to case in the manage services. Let’s assume that we have some environment variables that other people or application developers to use. So, we can create a User defined service and provide your variables through it. It works just like the manage services, so you can manifest to bind it to application or simply go to dashboard and bind it manually with the application.

Multiple Deployment Options:

There are couple of deployment options we have in Cloud Foundry which we can make use of it. So for service to be available for binding on the requirement is that, it should implement Service Broker API.

So assume that, we want to have a service definition, which goes into category of user defined service. We can simply attach that service as a resource to our application or for the other service deployment options, we can have our service defined and deployed in Cloud Foundry. Cloud Foundry platform is really flexible allowing use to know how we want it.

Previous: Cloud Foundry – Push Application into Space

Next: Create Service Instance in PCF