Welcome back.

This is part two of this lesson.

We're going to continue immediately from the end of part one.

So let's get started.

Now let's look at another example.

This looks more complex, but we're going to use the same process to identify the correct answer.

So this is a multi-select question and we're informed that we need to pick two answers, but we're still going to follow the same process.

The first step is to check if we can eliminate any of the answers immediately.

Do any of the answers not make sense without reading the question?

Well, nothing immediately jumps out as wrong, but answer E does look strange to me.

It feels like it's not a viable solution.

I can see the word encryption mentioned and it's rare that I see lambda and encryption mentioned in the same statement.

So at this stage, let's just say that answer E is in doubt.

So it's the least preferred answer at this point.

So keep that in your mind.

It's fine to have answers which you think are not valid.

We don't know enough to immediately exclude it, but we can definitely say that we think there's something wrong with it.

Given that we need to select two answers out of the five, we don't need to worry about E as long as there are two potentially correct answers.

So let's move on.

Now, the real step one is to identify what matters in the question text.

So let's look at that.

Now, the question is actually pretty simple.

It gives you two requirements.

The first is that all data in the cloud needs to be encrypted at rest.

And the second is that any encryption keys are stored on premises.

For any answers to be correct, they need to meet both of these requirements.

So let's follow a similar process on the answer text first looking for any word fluff and then looking for keywords which can help identify either the correct answers or more answers that we can exclude.

So the first three answers, they all state server side encryption, but the remaining two answers don't.

And so the first thing that I'm going to try to do with this question is to analyze whether server side encryption means anything.

Does it exclude the answers or does it point to those answers being correct?

Well, server side encryption means that S3 performs the encryption and decryption operations.

But depending on the type of server side encryption, it means that S3 either handles the keys or the customer handles the keys.

But at this stage, using server side encryption doesn't mean that the answers are right or wrong.

You can use it or you can't use it.

That doesn't immediately point to correct versus incorrect.

What we need to do is to look at the important keywords.

Now, if we assume that we are excluding answer E for now unless we need it, then we have four different possible answers, each of which is using a different type of encryption.

So I've highlighted these.

So we've got S3 managed keys, SSE-S3, KMS managed keys, which is SSE-KMS, customer provided keys, which is SSE-C, and then using client side encryption.

Now, the first requirement of the question states encryption at rest and all of the answers A, B, C and D, they all provide encryption at rest.

But it also states that encryption keys are to be stored on premises.

Answers A and B use server side encryption where AWS handle the encryption process and the encryption keys.

So SSE-S3 and SSE-KMS both mean that AWS are handling the encryption keys.

And because of this, the keys are not stored on premises and so they don't meet the second criteria in the question.

And this means that they're both invalid and can be excluded.

Now, this leaves answers C, D and E, and we already know that we're assuming that we're ignoring answer E for now and only using it if we have to.

So we just have to evaluate if C and D are valid.

And if they are, then those are the answers that we select.

So SSE-C means that the encryption is performed by S3 with the keys that the customer provides.

So that works.

It's a valid answer.

So that means at the very least C is correct.

It can be used based on the criteria presented by the question.

Now, answer D suggests client side encryption, which means encrypting the data on the client side and just passing the encrypted data to S3.

So that also works.

So answers C and D are both potentially correct answers.

So both answers C and D do meet the requirements of the question.

And because of this, we don't have to evaluate answer E at all.

It's always been a questionable answer.

And since the question only requires us to specify two correct answers, we can go ahead and exclude E.

And that gives us the correct answers of C and D.

So that's another question answered.

And it's followed the same process that we used on the previous example.

So really answering questions within AWS is simply following the same process.

Try and eliminate any crazy answers.

So any answers that you can eliminate based just on the text of those answers, then exclude them right away because it reduces the cognitive overhead of having to pick between potentially four correct answers.

If you can eliminate the answers down to three or two, you significantly reduce the complexity of the question.

The next step is to find what really matters in the question, find the keywords in both the preamble and the question.

Then highlight and remove any question fluff.

So anything in the question which doesn't matter, eliminate any of the words which aren't relevant technically to the product or products that you select.

So this is something that comes with experience, being able to highlight what matters and what doesn't matter in questions.

And the more practice that you do, the easier this becomes.

Next, identify what really matters in the answers.

So again, this comes down to identifying any shared common words and removing those and then identifying any of the keywords that occur in the answers.

And then once you've got the keywords in the answers and the keywords in the questions, then you can eliminate any bad answers that occur in the question.

Now, ideally at this point, what remains are correct answers.

You might start off with four or five answers.

You might eliminate two or three.

The question asks for two correct answers.

And that's it.

You've finished the question.

But if you have more answers than you need to provide, then you need to quickly select between what remains and you can do that by doing this keyword matching.

So look for things which stand out.

Look for things which aren't best practice according to AWS.

Look for things which breach a timescale requirement in the question.

Look for things which can't perform at the levels that the question requires or that cost too much based on the criteria and the question.

Essentially, what you're doing is looking for that one thing that will let you eliminate any other answers and leave you with the answers that the question requires.

Generally, when I'm answering most questions, it's a mixture between the correct answer jumping out at me and eliminating incorrect answers until I'm left with the correct answers.

You can approach questions in two different ways.

Either looking for the correct answers or eliminating the incorrect ones.

Do whichever works the best for you and follow the same process throughout every question in the exam.

The one big piece of advice that I can give is don't panic.

Everybody thinks they're running out of time.

Most people do run out of time.

So follow the exam technique process that I detailed in the previous lesson to try and get you additional time, leave the really difficult questions until the end, and then just follow this logical process step by step through the exam.

Keep an eye on the remaining amount of time that you have at every point through the exam and I know that you will do well.

Most people fail the exam because of their exam technique, not their lack of technical capability.

With that being said, though, that's everything I wanted to cover in this set of lessons.

Good luck with the practice tests.

Good luck with the final exam.

And if you do follow this process, I know that you'll do really well.

With that being said, though, go ahead and complete this video and then when you're ready, I'll look forward to joining you in the next.

Welcome back and from the very start this course has been about more than just the technical side.

So this is part one of a two-part lesson set and in this lesson I'll focus on some exam technique hints and tips that you might find useful in the exam.

Now in terms of the exam itself it's going to have questions of varying levels of difficulty and this is also based on your own strengths and weaknesses.

Conceptually though understand that on average the AWS exams will generally feel like they have 25% easy questions, 50% medium questions and 25% really difficult questions.

Assuming that you've prepared well and have no major skill gaps this is the norm.

For most people this is how it feels.

The problem is the order of the difficulty is going to feel random so you could have all of your easy ones at the start or at the end or scattered between all of the other questions and this is part of the technique of the AWS exams how to handle question difficulty in the most efficient way possible.

Now I recommend conceptually that my students think of exams in three phases.

You want to spend most of your time on phase two.

So structurally in phase one I normally try to go through all of the 65 questions and identify ones that I can immediately answer.

You can use the exam tools including mark for review and just step through all of the questions on the exam answering anything that's immediately obvious.

If you can answer a question within 10 seconds or have a good idea of what the answer will be and just need to consider it for a couple more seconds this is what I term a phase one question.

Now the reason that I do these phase one questions first is that they're easy they take very little time and because you know the subject so well you have a very low chance of making a mistake.

So once you've finished all of these easy questions the phase one questions what you're left with is the medium or yellow questions and the hard or red questions.

My aim is that I want to leave the hard questions until the very end of the test.

They're going to be tough to answer anyway and so what I want to do at this stage in phase two is to go through whatever questions remain so whatever isn't easy and I'm looking to identify any red questions and mark them for review and then just skip past them.

I don't want to worry about any red questions in phase two.

What phase two is about is powering through the medium questions.

These will require some thought but they don't scare you they're not impossible.

The medium questions so the yellow questions should make up the bulk of the time inside the exam.

They should be your focus because these are the questions which will allow you to pass or fail the exam.

For most people the medium questions represent the bulk of the exam questions.

Generally your perception will be that most of the questions will be medium.

There'll be some easy and some hard so you need to focus in phase two which represents the bulk of the exam on just these medium questions.

So my suggestion generally is in phase two you've marked the hard questions for review and just skipped past them and then you focused on the medium questions.

Now after you've completed these medium questions you need to look at your remaining time and it might be that you have 40 minutes left or you might only have four minutes or even less.

In the remaining time that you have left you should be focusing on the remaining red questions the difficult questions.

If you have 40 minutes left then you can take your time.

If you have four minutes you might have to guess or even just click answers at random.

Now both of these approaches are fine because at this point you've covered the majority of the questions.

You've answered all of the easy questions and you've completed all of the medium questions.

What remains are questions that you might get wrong regardless but because you've pushed them all the way through to the end of your time allocation whether you're considering them carefully and answering them because you have 40 minutes left or whether you're just answering them at random they won't impact your process in answering the earlier questions.

So if you don't follow this approach what tends to happen is you're focusing really heavily on the hard questions at the start of the exam and that means that you run out of time towards the end but if you follow this three-stage process by this point all that you have left is a certain number of minutes and a certain set of really difficult questions and you can take your time safe in the knowledge that you've already hopefully passed to the exam based on the easy and medium questions and the hard ones as simply a bonus.

Now at a high level this process is designed to get you to answer all of the questions that you're capable of answering as quickly as possible and leave anything that causes you to doubt yourself or that you struggle with to the end.

So pick off the easy questions, focus on the medium and then finish up with the really hard questions at the end.

I know that it sounds simple but unless you focus really hard on this process or one like it then your actual exam experience could be fairly chaotic.

If you're unlucky enough to get hard questions at the start and you don't use a process like this it can really spoil your flow.

So before we finish this lesson just some final hints and tips that I've got based on my own experiences.

First if this is your first exam assume that you're going to run out of time.

Most people enter the exam not having an understanding of the structure and most people myself included with my first exam will run out of time.

The way that you don't run out of time and the way that you succeed is to be efficient, have a process.

Now assuming that you have the default amount of time you need to be aware that you have two minutes to read the question, read the answers and to make a decision.

So this sounds like a lot but it's not a lot of time to do all of those individual components.

You shouldn't be guessing on any answers until the end.

If you're guessing on a question then it should be in the hard question category and you should be tackling this at the end.

I don't want you guessing on any easy questions or any medium questions.

If you're guessing then you shouldn't be looking at it until right at the very end.

Another way of looking at this is if you are unsure about a question or you're forced to guess early on you need to be aware that a question that's later on so further on in the exam might prompt you to remember the correct answer for an earlier question.

So if you do have to guess on any questions then use the mark for review feature.

You can mark any question that you want for review as you go through the course and then at any point or right at the end you can see all the questions which are flagged for review and revisit them.

So use that feature it can be used if you're doubtful on any of the answers or you want to prompt yourself as with the hard questions to revisit them toward the end of the exam.

Now this should be logical but take all the practice tests that you can.

One of my favorite test vendors in the space is the team over at TutorialsDojo.com.

They offer a full range of practice questions for all of the major AWS exams so definitely give their site a look.

One of the benefits of the exam questions created over at TutorialsDojo is that they are more difficult than the real exam questions so they can prepare you for a much higher level of difficulty and by the time you get into the exam you should find it relatively okay.

So my usual method is to suggest that people take a course and then once they've finished the course take the practice test in the course, follow that up with the tutorials Dojo practice tests and for any questions they get wrong it can identify areas that they need additional study.

So rinse and repeat that process, perform that additional study, redo the practice tests and when you're regularly scoring above 90% on those practice tests then you're ready to do the real exam.

And at this point there are all of my suggestions for exam technique.

In the next lesson I want to focus on questions themselves because it's the questions and your efficiency during the process of answering questions which can mean the difference between success and failure.

So go ahead complete this video and in the next video when you're ready we'll look at some techniques on how you can really excel when tackling exam questions.

Welcome back.

In this video, I want to cover another part of the AWS global network, specifically the Edge Network, and that's AWS Local Zones.

Now, this is a key architectural concept that you'll need to understand for all of the AWS exams, and especially so for the real world.

So let's jump in and get started.

Now, before we talk about local zones, let's just refresh our memory on what the typical region and availability zone architecture looks like without local zones.

So we have a region, and let's say that this is US West 2, and within this we have three availability zones, US West 2A, US West 2B, and US West 2C.

And then running in this region across those availability zones is a VPC.

Now, an AWS region has high performance and resilient internet connections, and sitting between these and the AWS private zone is the AWS public zone.

So this is the zone where all of the AWS public services for that region run within.

And then lastly, on our right, we have our business premises.

What we know about this architecture so far is that it scales.

It can grow with your requirements, and that's really important because this is fully managed within the region.

We also know that it's resilient to failure.

The failure of one availability zone won't impact other availability zones, assuming a solutions architect has designed a solution which has infrastructure duplicated across all of the availability zones and things in one availability zone consume from that availability zone only, often regionally resilient services.

Now, what I haven't talked about until now is the effects of geographic distance.

The availability zones in this region might be hundreds of kilometers away from the business premises.

Now, this distance, even assuming that we're using fiber, can cause latency.

And this latency causes a reduction in performance, and this performance impact is noticeable at this distance.

To many use cases, a few milliseconds of latency might not sound like much, but for applications which are sensitive to latency, this can really matter.

An example might be a financial trading application.

Even if we use Direct Connect, physics and the speed of data transfer from point A to point B matters.

So how can we fix this?

Well, we can use AWS local zones and let's see how this changes the architecture.

Let's adjust the diagram a little and make it easier to see.

And we're going to add some subnets in availability zone 2A, 2B and 2C.

And we'll also have some EC2 instances running in these subnets.

When we're discussing local zones, we can refer to this region as the parent region.

So this region is the parent region to any local zones which operate in the same geographic area.

So we're also going to add some local zones to this architecture.

Now, these are identified starting with the region name and then a unique identifier for the local zone.

In this example, we have US West 2 and then LAS-1, which is a local zone in Las Vegas.

And we have US West 2 as its parent region.

So you can see the link between the local zone and the parent region because you can read the parent region at the start of the local zone name.

Now, it's possible to have multiple local zones in a given city.

For instance, in this example, we have US West 2-LAS-1A and 1B.

And both of these are in Los Angeles.

Notice how they use the international city code to identify them.

Now, think of these as related to the parent region, but they operate as their own independent infrastructure points.

So they have their own independent connections to the internet.

And additionally, generally, they also support Direct Connect, which means you can achieve high performance, private connectivity between your business locations and these local zones.

Now, different services support local zones in different ways.

And over the course of your studies, you're going to learn how.

With EC2 and VPCs, the VPC is simply extended by creating subnets within the local zones.

And then within these subnets, you can create resources as normal, utilizing the proximity of the local zone.

So these resources benefit from super low latencies.

The performance between the business premises and the local zone is at the extreme end of what's possible because of the smaller geographic separation between the local zone and your business premises.

Now, an important thing to keep in mind is that some things within the local zones still utilize the parent region.

So in this example, the subnets created in the local zones behave just like those in the parent region, and they have private connectivity just like any other subnets would.

Local zones have private networking with the parent region.

So remember that.

However, if we create EBS snapshots, then these use S3 in the parent region.

It means they still benefit from the AZ replication across all availability zones within that region that snapshots would normally benefit from.

So certain things occur within the local zone, but certain things rely on the parent region.

And one common example is EBS snapshots.

Now, let's finish up this video with some key summary points because for most of the AWS certifications, you only need to have this high level architectural overview.

So think about local zones as one additional zone or one additional availability zone so they don't have built-in resilience.

Conceptually, one zone runs in one specific facility.

So you can think of them like a single availability zone but near your location.

So they're closer to you so they have lower latency and lower latency means better performance.

So just imagine taking one of the availability zones within a region and duplicating it but putting it in a building next to your business premises.

Now, it won't always be that close, but there are some businesses which are built very close to these AWS local zones by design.

So you're able to get really close to the AWS infrastructure.

Now, not all AWS products support using local zones and for the ones that do, many of them are opt-in and many of them have limitations.

So if you're ever going to utilize local zones, you need to make sure that you check the AWS documentation for an up-to-date overview of what's supported within the local zones in your specific geographic area.

And I've made sure to include a link attached to this video which gives you up-to-the-minute overviews for all of the AWS local zones.

Now Direct Connect to local zones is generally supported and this allows local zones to be used to support any extreme performance needs or performance requirements.

And once again, local zones do utilize the parent region for various things and one example is EBS snapshots are taken to the parent region and replicated over S3 in that parent region.

Now, just to summarize this, you should use local zones as an architect when you need the absolute highest level of performance.

Local zones, much like cloud front edge locations, are much more likely to be positioned closer to your business than the parent region and any of the normal availability zones.

But if you do utilize local zones, you need to make sure that they do offer the functionality that you require.

So essentially, this is just another tool that you can use to build architectures as a solutions architect.

Now this is everything I wanted to cover in this video.

I just wanted to give you a high level overview of the architecture of local zones.

So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this video, I want to cover Amazon SageMaker from a high level perspective.

Now this is a product which you need to understand at only a foundational level for most AWS exams, but in depth for some others.

If any additional knowledge is required for the course that you're studying, there will be follow up deep dive videos.

But if you only see this one, this is all that you'll require.

Now I have to apologize in advance.

I don't like creating lessons which I don't think add much real world value.

SageMaker is a special kind of product where you're only going to be able to use it effectively if you have some practical experience.

But because you don't need to understand how to use it in depth for most of the AWS exams, I don't really want to waste your time going into significant depth.

It's a fairly niche product to use in the real world.

So this lesson is really just going to present some high level features.

And you're not going to get as much value doing it this way.

But it's just one of those things that we have to do in this way to avoid wasting time.

So you probably won't really like this lesson, but just stick with it because it will benefit you for the exam.

Now let's jump in and get started straight away.

So SageMaker is actually a collection of other products and features all packaged together by AWS.

And it's an implementation of a fully managed machine learning service.

So it essentially helps you with the process of developing and using machine learning models.

So this includes data fetching, cleaning, preparing, and then training and evaluating models and then deploying those models and then monitoring those models and collecting data.

So it's used for this entire machine learning lifecycle and it's AWS's way of implementing one product or one container of products which can help you for this entire lifecycle.

Now there are a few key things that you need to be aware of about SageMaker.

The first is SageMaker Studio.

And this is essentially an IDE or integrated development environment for the entire machine learning lifecycle.

So this allows you to build, train, debug, and monitor machine learning models.

So think about this as a development environment for the machine learning lifecycle.

If you see this mentioned on the exam, you understand that at a high level what it is.

Now within SageMaker, you have the concept of a SageMaker domain.

And I want you to think about this as isolation or groupings for a particular project.

So you're provided with an EFS volume, separate users, you can use applications, policies, and different VPC configurations.

So just think of this as almost a container for a particular project.

And when you start using SageMaker, you'll have to create a SageMaker domain in order to interact with the product.

Next we've got containers.

And these are essentially Docker containers which are deployed to specific machine learning EC2 instances.

So these are specific types and sizes of EC2 instances which start with ML and they're designed specifically for machine learning workloads.

Now these Docker containers are machine learning environments which come with specific versions of the operating system, libraries, and tooling for the specific task that you're wanting to accomplish.

And there are many different pre-built containers that you can utilize with SageMaker.

Now SageMaker is also capable of hosting machine learning models.

So you can deploy machine learning models as endpoints that your applications can then utilize.

And there are various different hosting architectures which are either based on serverless or consistently running compute.

Now again, this is beyond the scope of this high level architecture video, but I did just want you to be aware that SageMaker can host your machine learning models.

This is something that you need to be aware of for the exam.

And then finally, be aware that SageMaker itself has no cost, but the resources that it creates do have a cost.

And it's fairly complex pricing because of the range of services which can be created by SageMaker.

And another important concern about SageMaker is because of the complexity of those resources and because of the compute requirements of machine learning in general, the resources which are deployed can be relatively large and carry significant cost.

And to help you with this, I'm going to include a link attached to this video which details some of the important cost elements for this product.

Now at this point, that's everything I'm going to cover in this video.

I need to apologize again, because I know it's at a very abstract and high level.

But the only thing that you need to be aware of for most of the AWS exams is how it works structurally at a high level.

There's only one exam where you need additional levels of understanding and that's the machine learning specialty exam.

So if you're studying this as part of the machine learning course, you're going to find many other videos which are deep diving into how SageMaker works, including advanced demos or mini projects which are going to give you practical experience.

But if you only see this one video, it means you're studying a course which only needs this really high level understanding.

And with that being said, that's everything I wanted to cover in this video.

So go ahead and complete the video.

And when you're ready, I look forward to you joining me in the next.

Welcome back and in this video I'm going to continue my super high level overview of the AWS machine learning services, this time looking at Amazon fraud detector.

Now once again for nearly all of the AWS certifications a very high level overview is more than enough.

If you do need any additional knowledge for the course that you're studying I will have follow-up videos but this one just covers the basics so let's jump in and get started.

Now Amazon fraud detector as the name suggests is a fully managed fraud detection service so this allows you to look at various historical trends and other related data and identify any potential fraud as it relates to certain online activities so this might include things like new account creations payments or guest checkouts.

Now the architecture of this much like the other managed machine learning services is that you upload some historical data and you choose a model type.

Now for this particular service we have a few different model types.

First we have online fraud and this is designed for when you have little historical data and you're looking to identify any problematic events such as new customer accounts.

You might not have any data for a particular customer logically enough they're just signing up for an account so this looks at general trends such as if there are any surrounding elements of concern around this particular sign-up for this particular user.

Now we also have another model type which is transaction fraud and this is ideal for when you do have a transactional history for that customer and this transactional history can be used to identify any suspect payments.

This is fairly commonly used when you're performing credit card transaction validation.

Generally you do have a full purchase history for a customer so for example what stores are used, the types of purchases that are used, the value of those purchases, the countries that the purchaser is in or the store is in as well as the amounts and times of day.

So you can build up a fairly good profile for a given customer of what their normal transactions are like and this is what this particular model allows so you can import the transactional history for one or more customers and use this to identify any suspect payments and then the last is the account takeover model type and this can be used to identify any phishing or other social media or social based attacks.

For example if somebody signs in from a completely different location or if they're referred from a certain site all of this can be taken into account to identify if this user has potentially been affected by an account takeover attempt.

Now the way that this product works is that all of the various events are scored and then you can create rules which is basically a type of decision logic and use this to react to these scores based on your business activity or business risk.

So Amazon Fraud Detector is another back-end style service which generally you're going to be integrating into your applications or environments rather than using interactively from the console.

Now this high-level understanding is everything that you'll need for most of the AWS exams so I'm going to limit this video to covering just this high-level overview.

So at this point we have finished go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video I'm going to continue my super high-level overview of the AWS machine learning services.

This time looking at Amazon forecast.

Now this is not weather forecasting but forecasting based on time series data.

Once again for nearly all of the AWS certifications a very high-level overview is more than enough.

If you need any additional knowledge I'll have follow-up videos but this one just covers the basics.

So let's jump in and get started.

So Amazon forecast provides forecasting for time series data.

Now this means things like predicting retail demand, supply chain, staffing levels, energy requirements, server capacity and web traffic.

So any type of data which is time series where you have a large amount of historical and related data well you can use this together with forecast to provide the ability to forecast future trends and events.

Now the way that you do this is you import historical data and related data.

Now simple historical sales data might include just an item being sold and then a date and timestamp and you can use this to trend how popular that item is throughout that time series and use that for simple forecasting.

Related data includes extra contextual information such as any promotions which might have been running throughout a time period and even things like the weather which can influence the data over a particular time period.

So forecast uses both of these different sets of data so historical and related and it understands what's normal over a particular time period and the output of this will be a forecast and forecast explainability.

Now the forecast is simple enough to understand.

It allows you to trend out the future demand for a particular thing or understand the future requirements for a particular thing.

The explainability though goes into more depth.

It allows you to extract the reasons for changes in this demand.

So for example I mentioned the weather as a related data item.

Well in the retail world weather can have a huge impact on the general demand levels for products.

Generally all products will be influenced to base level based on the weather.

Some weather patterns may cause people to stay at home versus some may encourage people to go shopping.

Now this obviously has a much greater effect for physical shopping in a retail establishment versus online but other elements of weather can also affect online shopping.

For example if it's raining heavily you might see a demand increase for certain types of clothing.

So this is all involved in the ability to forecast future demand based on historical data and this is what this product provides.

It's a managed service which provides the ability to get access to forecasting for time series data.

Now you can interact with the product from the web console where you can use it to see visualization.

So see past data, future forecasting as well as explore explain ability of that forecast.

It can also be interacted with using the CLI, APIs and the Python SDK.

Now once again this is a back-end service.

It's generally something that you're going to use as part of either a business process or integrating it with your application.

It's relatively niche in its use and so for most AWS exams this high-level overview is everything that you'll need.

If you do need any additional knowledge for the course that you're studying there will be additional theory and/or practical videos but at this point that's everything I wanted to cover in this high-level video.

Go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video, I want to cover the high level architecture of the Amazon Translate product.

This is another machine learning product available within AWS and if you need any other knowledge over and above architecture, there will be additional videos following this one.

If you only see this video, don't worry, it just means that this is the only knowledge that you need.

Now let's just jump in and get started straight away.

Amazon Translate as the name suggests is a text translation service which is based on machine learning.

It translates text from a native language to other languages one word at a time.

Now the translation process is actually two parts.

We first have the encoder and the encoder reads the source text and then outputs a semantic representation which you can think of as the meaning of that source text.

Remember that the way that you convey certain points between languages differs.

It's not always about direct translation of the same words between two different languages.

So the encoder takes the native source text and it outputs a semantic representation or meaning and then the decoder reads in that meaning and then writes to the target language.

Now there's something called an attention mechanism and Amazon Translate uses this to understand context.

It helps decide which words in source text are the most relevant for generating the target output and this ensures that the whole process correctly translates any ambiguous words or phrases.

Now the product is capable of auto-detecting the source text language.

So you can explicitly state what language the source text is in or you can allow that to be auto-detected by the product.

Now in terms of some of the use cases for Amazon Translate, well it can offer a multi-lingual user experience.

So all the documents that exist within businesses are generally going to be stored in the main language of that business.

But this allows you to offer those same documents such as meeting notes, posts, communications and articles in all of the languages that staff within your business speak and this can make it much easier for organizations which have officers in different countries to operate more efficiently.

This also means that you can offer things like emails, in-game chat or customer live chat in the native language of the person that you're communicating with and this can increase the operational efficiency of your business processes.

It also allows you to translate incoming data such as social media, news and communications from the language that they're written in into the native language of the staff that are interpreting those incoming communications.

Now more commonly, Amazon Translate can also offer language independence for other AWS services.

So you might have other services such as Comprehend, Transcribe and Poly which operate on information and Translate offers the ability for these services to operate in a language independent way.

It can also be used to analyze data which is stored in S3, RDS, DynamoDB and other AWS data stores.

Now generally with this product you're going to find that it's used more commonly as an integration product.

So rather than use it directly, it's more common to see it integrated with other AWS services, other applications including ones that you develop and other platforms.

So in the exam if you see any type of scenario which requires text to text translation then think Translate.

If you see any scenario which might need text to text translation as part of a process then Translate can form part of that process.

So you might want to translate one language into another and then speak that language or you might want to take audio which is in one language output text and then translate that to a different textual language.

Keep in mind that Translate is often used as a component of a business process.

So really keep that one in mind.

It's not always used in isolation.

Now with that being said, that is everything I wanted to cover in this video.

So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this video we're going to be doing a really quick overview of Amazon Transcribe.

Now there isn't a lot to understand about this product so let's just jump in and get started straight away.

Amazon Transcribe is an automatic speech recognition or ASR service.

Essentially the input to this product is audio and the output is text.

So it takes audio in the form of speech and then outputs the text version of that speech.

And it offers various features which improve this process so things like language customization, filters for privacy, audience appropriate language as well as speaker identification.

In addition you can configure custom vocabularies as well as creating language models specific to your use case.

Now the product is actually pay-per-use and your build per second of transcribed audio.

Now with this product it's going to be much easier to see it working so I'm going to switch across to my console and give you a quick demonstration.

Okay so I'm at the AWS console, I'm logged in as the I am admin user of my general AWS account and I'm in the northern Virginia region.

I'm just going to go ahead and click in the search box at the top and type transcribe and then click to move to the transcribe console.

Once I'm here I'm going to click on real-time transcription because this is what I'm going to use to demo the product.

And once I'm here I'm going to click on start streaming, say something into my microphone and then click on stop streaming.

I like cats, dogs, chickens and rabbits.

Spiders not so much.

At this point we can see that the product has transcribed what I said into my microphone into text and then by clicking here we could download a full transcript of this audio.

Now if we just scroll down we're able to look at some of the more advanced features of the product so you can specify a specific language or you can set it to auto detect the language.

You're able to enable or disable speaker identification, set various content removal settings such as vocabulary filtering or personally identifiable information and other redactions.

If we expand customizations it's here where we can specify custom vocabulary, partial result stabilization or a custom language model and we can even expand this to look at application integration.

So this product is capable of being integrated with either your own applications or other AWS products and it's on the menu on the left where you can interact with the specific versions of Amazon Transcribe such as call analytics or Transcribe Medical.

Now both of these are beyond the scope of what I want to cover in this video but I did want you to be aware that they exist.

Now at this point that's everything that I wanted to do in the walkthrough so I'm going to move back to the second part of the theory I'll be covering in this video.

Now in terms of the use cases of Amazon Transcribe it allows you to do full text indexing of audio so you can convert audio into text and that can be used for full searching of the text version of that audio.

It can also be used to create meeting notes.

If you use the product to ingest audio from any meetings that you conduct then you can have full text records of those meetings.

It can be used to generate subtitles, captions or transcripts of any video that your organisation uses.

The product comes in a number of specific flavours so Transcribe, Transcribe Medical and then Transcribe Call Analytics and in the call analytics version you're able to ingest audio from any audio phone calls and assess the characteristics, perform summarisation, look at categories and sentiments of the people talking on that phone call.

The product is also capable of being integrated with your own applications through APIs as well as being able to be integrated with other AWS machine learning services so Amazon Transcribe can convert from speech into text and then that text can be ingested by other AWS machine learning products to perform further analysis.

Now that's the product at a high level, in this video we're just going to stick to this high level summary.

If the topic that you're studying requires any additional information there will be additional theory and if required practical videos.

But for most of the AWS certifications this is all the information that you'll require so go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video, I want to briefly talk about Amazon Textract.

Now much like the other machine learning products which I'll be covering, in this video I'll only be covering Textract from a high level architecture perspective.

If more knowledge is required, there will be additional theory or practical videos, but don't be alarmed if this is the only one.

That just means that this is the only level of knowledge that you'll require.

Now let's jump in and get started straight away.

So Amazon Textract is another machine learning product available within AWS.

And this product is used to detect and analyze text contained within input documents.

And currently input documents mean JPEGs, PNG, PDF or TIFF files.

Now these are the inputs and then the outputs is extracted text, the structure of that text and then any analysis which can be performed on that text.

So this is a product which can take in one of the supported document formats and extract any relevant information.

And as you'll see, this can include things like generic documents, identity documents or receipts or invoices.

And I'll be demoing a couple of these from within the AWS console later on in this video.

Now for most documents, the product is capable of operating in a synchronous way, so real time.

So if you're inputting a normal size document, then you can expect the results of that analysis almost immediately after you submit.

For large documents, these are processed in an asynchronous way.

So this might be a large PDF, I think hundreds of pages.

And for this, you might have to submit the job and then wait for processing to occur.

Now the product is paper usage, but it does offer custom pricing available for large volumes.

Now in terms of the use cases of this product, at a high level, it offers the detection of text and the relationship between that text.

So you might have, for example, a receipt or invoice and it will be able to detect all of the relevant items, so prices and products, dates, as well as any interaction between those different elements.

So it might know, for example, that a particular product line has a specific price and this has a specific element of tax.

It also generates metadata.

So for example, where that text occurs.

And then for particular types of documents, it offers specific types of analysis.

So for generic documents, it might be able to identify names, addresses and birth dates, but then for receipts, it might be able to identify prices, vendors, line items and dates.

For identity documents, it can also do abstraction of certain fields.

So you might have driver's licenses, which offer a driver license ID and then passports, which offer passport IDs.

And the product is capable of assessing both of these and abstracting that in to a document ID field.

So you can analyze many different types of identity documents and store all of that data in a common database schema, which has abstracted field IDs such as document ID.

Now, again, this product is capable of either being used from the console UI or from the APIs.

And so it can be integrated with any applications that you either develop or architect.

As well as this, it can also be integrated with other AWS products and services, including other machine learning products.

Now it's going to be far easier for you to be able to understand how this product works if I give you a brief example.

So I'm going to go ahead and switch across to my AWS console and step through a couple of examples.

So let's go ahead and do that.

Okay, so I'm at the AWS console and logged into the IAM admin user at the general AWS account.

And I have the Northern Virginia region selected.

So I'm just going to go ahead and click in the search box at the top and type text tract.

And then click to move to the text tract console.

And I'm just going to step through a very simple set of examples.

So I'm going to go ahead and click try Amazon text tract and the console itself will give you a couple of examples of how it can be used.

So in this particular case, it's analyzed a vaccination card.

So you can see that on this card, the data is in both a structured and unstructured format.

So we have a number of the fields that are stored in a relatively structured way.

So we have last name, first name, date of birth and a patient number.

We also have a table below it, which stores the dates of the various vaccinations.

And as you might know, if you've taken a number of vaccinations, this can either be typed, handwritten or stamped.

And it's often not always within the nice confines of a particular area of the table.

Now this product is able to extract all of the important elements.

You can see that as I'm hovering my mouse here, it's able to identify the particular elements of data.

So I can click on all of these individually.

And you can see that they're now selected in this results table on the right.

So it's intelligent enough to identify these individual elements, even if they're slanted or different sized or in a hard to read format.

So this is a simple vaccination record.

And we can see all of the information stored on the right.

We do have different sample documents though, if we click on this dropdown, we've got a pay stub.

This is obviously more complex with a larger amount of data.

And as you can see, it's still identified all of this information.

So it's extracted all of the key values from this table.

Now what's even cooler is if I click on tables on the right, we can see that for this specific table, it's even got the intelligence, not only to extract the data, but also to extract the actual table structure itself.

I can move through the document and see multiple tables, each containing information.

And as you can see, a lot of these are really complex in the formatting.

And yet the product has managed to extract all of the data with no problems.

We've got other types of documents.

So this, for example, is a loan application.

And again, the text extract product is capable of extracting all of this, including larger blocks of text and enable us to browse through and deal with that data, both interactively from the console, but we could also do this from the APIs from within our own application.

Now the product also has some specific types of analysis that it can do.

So if I click on analyze expense, it's able to perform an extraction of data on a sample receipt.

So you can see in this example, not only is it able to extract textual data, but it can also extract vendor information from these receipts.

We can also perform the same type of process to analyze ID documents.

And in this case, we're showing a driver's license and it's able to extract this number on this driver's license to be a document number.

So we can see this document number here.

It's abstracted this driver's license number to be the document number.

And if we switch to a passport document, in this particular case, it's going to take the passport number and it will extract this also into a document number.

Now, if you're doing identity document verification, for example, if you run an online application and need to perform know your customer or anti-money laundering techniques, so you need to identify and verify ID documentation, then this ability to take specific fields of certain identity documents and abstract them away and use the same data structure is really valuable.

Now, this is everything I wanted to cover about the Textract product.

Again, this is a basic architectural level introduction.

If for the particular topic that you're studying, you require additional information, then there will be videos following this one, which go into more depth, either theory or practical.

But if you don't see any additional videos on this product, don't worry, it simply means that for whatever topic you're studying, this is all the information you require.

At this point though, that is the end of this video, so go ahead and complete the video and when you're ready, I look forward to you joining me in the next.

Welcome back and in this video, I want to talk about another really cool AWS product called Amazon Recognition with a K.

Now let's jump in and get started because I'm actually super excited to step through this product and how it works.

Recognition is a deep learning based image and video analysis product.

Deep learning is a subset of machine learning.

So this is to say that recognition can look at images or videos and intelligently do or identify things based on those images or videos.

Specifically, it can identify objects in images and videos such as cats, dogs, chickens, or even hot dogs.

It can identify specific people, text, for example, license plates, activities, what people are doing in images and videos.

It can help with content moderation.

So identifying if something is safe or not.

It can detect faces.

It can analyze faces for emotion and other visual indications.

It can compare faces, checking images and videos for identified individuals.

It can do pathing, so identify movements of people in videos.

And an example of this might be post game analysis on sports games and much, much more.

It's actually one of the coolest machine intelligence services that AWS has.

And that's saying a lot.

The product is also pay as you use per image pricing or per minute of video.

And it integrates with applications via APIs and it's event-driven.

So it can be invoked say when an image is uploaded to an S3 bucket.

But one of its coolest features is that it can analyze live video by integrating with Kinesis video streams.

So this might include doing facial recognition on security camera footage for security type situations, distinguishing between the owner of a property and somebody who's attempting to commit a crime.

All in all, it's a super flexible product.

Now generally for all of the AWS exams, you will need to have a basic understanding of the architecture.

There are some AWS exams, for example, machine learning, where you might need additional understanding.

And if you're studying a course where that additional understanding is required, there will be follow-up videos.

In general though, it's only a high-level architecture understanding.

And one example architectural flow might look something like this.

So an image containing whiskers and woofy is uploaded to S3.

Now we've configured S3 events and so this invokes a Lambda function.

The Lambda function calls recognition to analyze the image.

It returns the results and then the Lambda function stores the metadata together with a link to the image into DynamoDB for further business processes.

To give some context as to the other things that recognition can do, let's just take an entirely random selection of images from the internet.

So recognition can identify celebrities such as Ironman.

It can also identify mic chambers that I think as a machine learning service, it might be slightly biased.

It can identify text in images or videos such as license plates on cars or other internet memes.

It can even identify objects, animals, or people in those same memes.

For faces specifically, it can identify emotions or other attributes.

So for example, identifying this random doctor is a male who currently has his eyes open and is looking very, very serious rather than being happy in any way.

So that's recognition.

If you have questions in the exam which need general analysis performing on images or videos for content, emotion, text, activities, anything I've mentioned in this lesson, then you should default to picking recognition.

It's probably going to be the correct answer.

Now with that being said, that is everything I wanted to cover in this video.

Go ahead and complete this video And when you're ready, I look forward to you joining me in the next.

Welcome back.

And in this video, I want to briefly talk about the Amazon Poly product.

Now, this is going to be another very brief video.

All you need to know for most AWS exams and to get started in the real world is the product's high level architecture.

If there are any other knowledge requirements for the topic that you're studying, there will be additional videos following this one.

If not, don't worry, this is everything that you'll need to understand.

Now, let's just jump in and get started.

So Amazon Poly at a high level converts text into life-like speech.

So in the example below, it takes I like cats, dogs, chickens and rabbits, spy does not so much.

It takes that text and it generates a life-like voice or life-like speech.

So essentially the product takes text in a specific language and results in speech, also in that specific language.

This is really important to understand.

Poly performs no translation.

It can only take text in a given language and output speech also in that language.

Now, there are two modes that Poly operates in.

We've got standard TTS or text to speech and this uses a concatenative architecture.

It essentially takes phonemes, which are the smallest unit of sound in the English language.

For example, for the letter A, the phoneme is a.

So it takes those smallest units of configurations and uses a concatenative architecture to build patterns of speech.

We've also got neural text to speech.

This takes those phonemes, it generates spectrograms, it puts these spectrograms through a vocoder and that generates the output audio.

Now, this is a much more complex way of generating speech using artificial intelligence.

It's much more computationally heavy, but what it does is result in much more human or natural sounding speech.

Now, the product is capable of outputting in different formats.

So you've got MP3, Agvorbis or PCM.

And the output format that you'll choose depends on how you're intending to integrate Poly with other products.

So you might choose PCM, for example, if you're wanting to integrate with various AWS products.

It fully depends on what your architecture is.

Now, there are a few final points that I want to cover.

First is that Poly is capable of using the speech synthesis markup language.

And this is a way that you can provide additional context within the text so you can control how Poly generates speech.

So examples of this include that you might want to get Poly to emphasize various parts of sentences or pronounce things in certain ways.

You might want Poly to whisper certain components of the text or you might want to use an over-exaggerated newscaster speaking style.

These are all things that you can control with speech synthesis markup language, which is SSML.

Now, again, Poly is the type of product that's going to be integrated with other things.

For example, you can get a WordPress plugin which allows articles on WordPress blogs to be spoken.

Poly can be integrated with other AWS services where you need speech to be generated based on text.

Or you can integrate Poly with your own applications using the APIs.

Again, this is another product which is going to be more often integrated with other things.

Now, this isn't a product where you're going to get much benefit from seeing this in action.

It's a very niche product that you're only ever going to use in certain situations.

For most AWS exams, you just need a high level architectural overview.

So at this point, that is everything I wanted to cover in this video.

Go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to cover the high level architecture of Amazon Lex and Amazon Lex is a product which allows you to create interactive chatbots.

Now for most areas of study and for solutions architects in the real world you just need to have a basic level of understanding and that's what this video will provide.

If you need to know anything else in the course that you're studying there's going to be follow-up videos to this one.

If not don't worry this video will cover everything that you need.

Now let's jump in and get started.

Amazon Lex is a back-end service.

It's not something that you're likely to use from a user perspective.

Instead you'll use it to add capabilities to your application.

So Lex provides text or voice conversational interfaces.

For the exam remember Lex for voice or Lex for Alexa.

If you're familiar with the Amazon voice products then just know that Lex powers those voice products so it provides the conversational capability.

It's what lets the lady in the tube answer your questions.

Now Lex provides two main bits of functionality.

First automatic speech recognition or ASR which is simple speech to text.

Now I say simple but doing this well is exceptionally difficult.

If any of you have tried using Siri which is Apple's voice assistant notice how often it gets things wrong versus the Alexa product.

That's because it doesn't do ASR as well as Lex.

And for any lawyers listing this is just by opinion.

Now Lex also provides natural language understanding or NLU services and this allows Lex to discover your intent and can do intent chaining.

So imagine the act of ordering a pizza.

How would you start off that conversation?

Maybe can I order a pizza please?

Or maybe I want to order a pizza.

What about a large pepperoni pizza please?

The intent the thing you want to do is order pizza.

It's Lex's job to determine that.

But what about your next sentence?

Make that an extra large please.

Well Lex needs to understand that the second statement relates to the first.

As a human it sounds easy because humans are good at this type of natural language processing.

Computers historically have not been so great.

So Lex allows you to build voice and text understanding into your applications.

It's the type of thing that you would use when you don't want to code the functionality yourself.

You integrate Lex and it does the hard work on your behalf.

Now as a service it scales well and integrates with other AWS products such as Amazon Connect.

It's quick to deploy and uses a pay-as-you-go pricing model so it only costs when using it.

It's perfect for event driven or serverless architectures.

Now in terms of the use cases which Lex can help with you might use it to build chat bots, those annoying apps on web pages which ask you if you want help or automated support chats when logging support tickets.

It can be used to build voice assistants, you ask for something and the lady in the tube delivers.

Things like QA bots or even info or enterprise productivity bots.

Any interactive bot which accepts text or voice and performs a service.

Now let's review some of the key Lex concepts.

So Lex provides bots and bots are designed to interactively converse in one or more languages.

I've mentioned the term intent previously.

This is an action that a user wants to perform.

So things like ordering a pizza, ordering a milkshake or getting a side of fries.

Now as well as the intent we also have the concept of utterances and when creating an intent you're able to provide sample utterances and utterances are ways in which an intent might be said.

So in order to order a pizza, a milkshake or some fries you might start off by saying can I order or I want to order or give me a.

These are all different ways, different ways you can utter or provide utterances for an intent.

In addition to configuring utterances you also have to tell Lex how to fulfill the intent and often this is using lambda integration.

So if Lex understands that you do want to order a pizza it needs to have some way of initiating that pizza ordering process and often this is using lambda.

Again lambda is really great in an event driven architecture so it integrates and complements Lex really well.

Additionally you also have the concept of a slot and you can think of these as parameters for an intent.

So you might have things like the size of a pizza, small, medium or large and what type of crust normal or cheesy and you can configure these to be required parameters that go along with an intent.

So pieces of information that Lex needs to gain from that interaction with a user and just to reiterate Lex is the type of product that you're not going to use directly via the console it's going to be something that you're going to architect or develop into your applications.

If you want to provide interactive voice assistance via a chat or voice capable bot then you're going to use Amazon Lex.

So remember this for the exam.

With that being said that is everything I wanted to cover in this video so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video, I'm going to quickly discuss the functionality provided by Amazon Kendra.

Now, this is something which you only need to have the highest level exposure to for most of the AWS exams.

If you need to know more than this basic level, there will be other videos diving deeper into Kendra functionality, but this video will stick to the basics.

So if it's the only video on Kendra in the course that you're taking, that's fine.

This is everything that you need to know.

So let's jump in and get started.

Now, Kendra is an intelligent search service.

So its primary aim is that it's designed to mimic interacting with a human expert.

So the idea is that you can use Kendra to search a source of data and you feel as though you're interacting with a human expert on that data subject.

So it supports a wide range of question types.

We start with simple factoid based questions, such as who, what and where.

So you can ask questions like who wrote this book, what the best type of animal is, of course it's cat, and where is this particular place.

And Kendra will attempt to surface the answer to that information and do it in such a way that it mimics the type of interaction and the quality of response that you get with a human.

Now, another type of question is a descriptive question.

And this might come in the form of how do I get my cat to stop being a jerk.

And so Kendra has to understand all the question text as well as the intent of the question.

So what it is you're trying to surface.

And then lastly, we have keyword type questions.

Now, these are much more difficult than they seem on first glance.

Imagine that you ask a question, what time is the keynote address?

Well, in this particular case, a dress doesn't always have the same meaning.

You can mean a postal address or you can mean in this context, a speech.

So a keynote address is a speech, a keynote speech of a conference.

And so Kendra has to help to determine the intent of the question being asked.

And that's why it's a very difficult problem to solve and why Kendra adds significant value by delivering this as a service.

Now Kendra is another backend style service.

You're going to use this product to provide functionality to any applications or systems that you design or implement.

So generally you will provide search capability within an application and use Kendra as a backend.

Now let's go through the key concepts of Kendra.

We start with an index, which is a searchable block of data organized in an efficient way.

So the index is the thing that Kendra searches when dealing with user queries.

We have a data source and this is the original location of where your data lives.

So Kendra connects and indexes from this location.

Now examples of this might be an S3 bucket, Confluence, Google Workspaces, RDS OneDrive, Kendra Webcrawler, WorkDocs, FSX, and many other data sources.

Now there are too many to list on this screen but I will attach a link to this video which gives an exhaustive list.

Now the idea is that you configure Kendra to synchronize a data source with an index based on a schedule.

And this keeps the index current with all of the data that's on the original data source.

Now in terms of the data that's being indexed, you have documents and these are either going to be structured or unstructured.

So structured might include things like frequently asked questions and unstructured data might be things like HTML files, PDFs and text documents.

But again, there are many more and there isn't space to have them on screen at the same time.

So I'll include a link attached to this video which gives a full overview of all of the different types of documents that Kendra can index.

Now Kendra as a product integrates with lots of different AWS services such as IAM for security and the IAM Identity Center previously known as AWS SSO for various single sign-on services.

And again, just to reiterate, Kendra is a backend product.

It's something that provides functionality to something else.

And so you're going to be interacting with Kendra using APIs which are surfaced through your applications.

It's not something that you'll generally interact with as an end user through the AWS console beyond the initial setup process.

Now that's everything that you need to understand for most AWS exams and to get started with it in the real world as a solutions architect.

So at this point, go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this video I want to very briefly talk about Amazon Comprehend and this is a natural language processing service available within AWS.

In short you input a document and it develops insights by recognizing the entities, keyphrases, language sentiments and other common elements of that document.

Now this video is going to cover the basics.

If you need to have a greater level of understanding for the thing that you're studying there will be follow-up videos.

But in this video we're going to cover the important high level elements so let's jump in and get started.

So I mentioned moments ago that Amazon Comprehend is a natural language processing service and that you input a document so conceptually think of this as text and then the output is any entities, phrases, language, personal identifiable information so any important elements of a document is surfaced by the Comprehend service and you get access to all of these things which have been identified.

Now the Comprehend service is a machine learning service and it's based on either pre-trained or custom models so you've got the option of using either of these.

Now the product is capable of doing real-time analysis for small workloads or asynchronous jobs for larger workloads and the service can be used from the console or command line for interactive use or you can use the APIs to build Comprehend into your applications.

Now it's going to be far better for you to gain an understanding of this product if you can see it operate visually.

So I'm going to go ahead and switch across to my console and just give a brief walkthrough of exactly what the product can do.

Now you can either watch me do this walkthrough or you can follow along within your own environment but I'm going to go ahead now and switch across to my AWS console.

Okay so I'm currently logged into my AWS console I'm logged in as the IAM admin user of the general AWS account and as always I've got the Northern Virginia region selected.

Now once I'm here I'm going to click in the search box at the top and just go ahead and type Comprehend and then click on Amazon Comprehend.

When you first move to the Comprehend service you'll see something like this you've got the hamburger menu on the top left where you can access all of the detailed areas of the console but just to illustrate this I'm going to go ahead and click on launch Amazon Comprehend.

Now when you first launch the product it does give you some sample input text so that you can see exactly how the product works and we're going to start with this sample text.

So if I just scroll down slightly under input text you'll be able to see the analysis type is built in and this allows you to have real-time insights based on the AWS built-in models so this is one of the machine learning models which I discussed in the theory part of this lesson.

We've also got custom and this allows you to create custom models which fit your data.

In this case though we're going to use the built-in analysis type and if I just scroll down this is sample input text it essentially describes an individual, a company, it includes some credit card information as well as other personally identifiable information.

So what I'm going to do is go ahead and click on analyze to analyze this sample input text.

So examples of what we get are entities so it searched the input text and it's identified anything which it classifies as an entity and examples of this are persons so the person entity and it identifies a person here which is the recipient of this email as well as the sender.

So John is identified as a person and it has a 0.99 plus confidence.

Now this is expressed from a value from 0 which is 0% through to 1 which is 100% and the confidence level shows how confident comprehend is of its identification.

In this case it's over 99% sure that John is a person.

Next we have any company financial services and this has been identified as an organization again with a high level of confidence.

We have a credit card information and this has been identified as an entity of type other.

We have a location so an address and then other again in the form of an email address again with 99% plus confidence.

Now what we can also do is click on key phrases and have it filter based on all of the key phrases within this text.

We can click on language and the product is capable of identifying any languages used in the text in this case English with again a 99% confidence.

We can click on PII and see any personally identifiable information.

So we have names, we have credit card numbers, date and times, bank account numbers, bank routing, address, phone numbers and emails and we can even click on sentiment to identify the overall level of sentiment in this text and for this it's identified the majority as being neutral.

Now what I could do is replace this text so I'm going to do that.

So I'm going to click in the box and delete it and I'm going to replace it with this.

So my name is Adrian and I'm 1300 and 37 years old, my favorite animals are cats and I own 500 of them and my least favorite are spiders.

I'm going to go ahead and analyze this piece of text.

This time what I want to focus on is the sentiment analysis so it still identifies all of the keywords, the quantities in the form of my age and the number of cats that I own.

It's still identifies the language used which is English and if I click on sentiment we'll see that there is some neutral, positive and negative but overall it's a mixed sentiment and that's because I specified that my favorite something is cats and my least favorite are spiders.

So there's a mixed level of sentiment in this text.

If for us to go ahead and delete the spiders part so just have my name, my age, my favorite animals and how many of them I own and then go ahead and analyze this and then go to sentiment.

Now we'll see that it's changed.

We have a mixture of neutral sentiment and positive sentiment so you can see how sentiment, key phrases and any personally identifiable information can be surfaced from an input document and that's what the Comprehend service does.

Now again this can be used from the console, the CLI and the API so while I'm demonstrating a very simple interaction with this product you can use the API's and integrate it with your applications or other parts of the system so this is an important product to understand both from a solutions architect, developer and operations perspective.

Now at this point that is everything I wanted to cover in this video I just wanted to give you a really high-level overview of how the Comprehend product works.

At this point though go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk in a little bit more detail about the backups and resilience options we have with Redshift.

In the last lesson you learned how Redshift functions within a single availability zone and so it's at risk from the failure of that availability zone.

Based on this as a solutions architect you need to ensure that you design your systems with this in mind.

So let's step through the options that we have and I'll try to keep this nice and brief because it's really a continuation of what I covered in the previous lesson.

Now let's say that we have a Redshift cluster in US East 1.

It's simplified a bit and so it only has two availability zones, Availability Zone A and Availability Zone B.

And as we know now Redshift runs from only one of them.

In this example Availability Zone B.

Now this means that if we have a failure in Availability Zone B we have problems.

The entire cluster will fail and any data it's managing will be at risk.

Redshift provides a number of useful recovery features.

First it can utilize S3 for backups in the form of snapshots.

Now there are two types of backups supported by the system.

Automatic backups which occur once every eight hours or so or every five GB of data change and these occur automatically into S3.

They have by default a one day retention period configurable for anything up to 35 days and you get backup capacity totaling the capacity of your cluster for free included in the price of the cluster.

Now the snapshots are incremental so only the data changed is stored and charged for.

There are also manual snapshots which are performed explicitly by a person or a script or a management application and these have no retention they last until they're removed by a manual process.

For both types of backups because they're stored on S3 you immediately benefit from the resilience profile of S3 meaning that data is replicated between three or more Availability Zones in that same region.

So while Redshift isn't resilient across Availability Zones in a region the data managed by Redshift can be.

Restoring from snapshots creates a brand new cluster and you can choose a working Availability Zone for that cluster to be provisioned into.

Now if you have a major problem in the region impacting multiple or all Availability Zones in that region then Redshift offers further resilience still.

You can configure snapshots to be copied to another AWS region.

For example you might choose to copy snapshots to AP, Southeast 2, Land of Kangaroos, Australia.

This means your data would be safe even from the failure of the entire original region and a new cluster could be provisioned in this new region quickly in the event of a disaster.

Copied snapshots can be set with an independent retention period and this can help minimize costs.

Now that's pretty much all I wanted to talk about with regards to Redshift backups.

There are questions in the exam about Redshift and high Availability and resilience options and so I think it's important to cover that and how you can avoid the single AZ risks by using backups effectively and so that's what we've covered in this lesson.

With that being said, thanks for watching, go ahead and complete the lesson and then when you're ready you can move on to the next.

Welcome back.

In this lesson, I want to briefly talk about Amazon Redshift.

Redshift is a complicated product and there's no way that I can talk about it all in this lesson.

So I'll be focusing on the things which really matter for the exam from a solutions architect perspective.

Now we have a lot to cover, so let's jump in and get started.

Redshift is a petabyte scale data warehouse.

A data warehouse is a location where many different operational databases from across your business can pump data into for long-term analysis and trending.

It's designed for reporting and analytics, not for operational style usage.

And I'll explain what that means in a second.

Now it's petabyte scale because it's been designed from the ground up to support huge volumes of data.

Now Redshift is an OLAP database rather than OLTP, which is what RDS is.

The difference is really important to understand for the exam.

Online transaction processing or OLTP captures, stores and processes data from transactions in real time.

So this is the type of database used when say adding orders to an online store or a database of the best cat pictures in the world.

It's designed as the name suggests for transactions and this means inserts, modifies and deletes.

Online analytical processing or OLAP is designed for complex queries to analyze aggregated historical data from OLTP systems.

So other operational or OLTP systems put their data into OLAP systems.

So RDS might put its data into Redshift for more detailed long-term analysis and trending.

Now Redshift stores its data in columns.

Imagine a database of all the best cats in the world.

Every row in the database represents one cat.

It stores the microchip ID, the name, the age, the color, its favorite food and so on.

In a row based or OLTP database, data is stored in rows because you always interact with specific records, specific cats.

So updating their ages or editing other attributes.

Now this means that if you wanted to query the average age of all the cats in the database, you'd need to read through every row looking for just one field.

With column based databases, data is stored in columns.

So all the names, all the ages and so on.

It makes reporting style queries much easier and more efficient to process.

Now Redshift is one such database.

It's a column based database and Redshift is delivered as a service just like RDS.

So it's pretty quick to provision and you can actually provision it, load data, use it for something and then tear it down when you finished.

Generally data is loaded into Redshift before being worked on, but the product includes some really advanced functionality.

Two in particular are really cool.

First is Redshift Spectrum, which allows for querying of data on S3 without loading it into Redshift in advance.

And this is great for larger datasets.

You still do need a Redshift cluster, but it means that instead of going through the time consuming exercise of loading data into the platform, you can use Redshift Spectrum to query data on S3.

There's also Federated Query, which is kind of like Federated Identity, but instead of Identities, it allows you to directly query data that's stored in remote data sources.

So essentially you can integrate Redshift with other databases, foreign or remote databases and query their data directly.

Now Redshift integrates with other AWS tooling such as QuickSight for visualization and it has a SQL like interface for data access.

It allows you to connect using JDBC and ODBC standards.

So if your database app supports either, then it can connect natively to Redshift.

Now let's go through some key architectural points before we look visually at how Redshift fits together.

First, Redshift is a provisioned product.

It uses servers, so it's not a serverless product like say Athena.

It's also not something that you would really use on an ad hoc basis like Athena.

It's much quicker to provision than on-premise data warehouse that you have to create yourself, but it does come with a provisioning time.

It's not something that should be used for ad hoc queries of large scale datasets on S3.

That's much more aligned to the functionality which Athena provides.

So for ad hoc queries, look towards Athena as a default rather than Redshift.

Now Redshift uses a cluster architecture and a really important architectural principle to understand is that the cluster is actually a private network.

You can't access most of the cluster directly.

Redshift runs with multiple nodes and high speed networking between those nodes.

And because of this, logically it runs in one availability zone.

So it's not highly available by design.

It's tied to a specific availability zone.

All Redshift clusters have a leader node and it's the leader node that you interact with.

The leader node manages communications with client programs and all communications with the compute nodes.

It develops execution plans to carry out complex queries.

So specifically about the compute nodes, these run the queries which were assigned by the leader node and they store the data loaded into the system.

A compute node is partitioned into slices.

Each slice is allocated a portion of the node's memory and disk space where it processes a portion of the workload assigned to that node.

The leader node manages distributing data to the slices and apportions the workload for any queries or other operations onto the slices.

The slices then work in parallel to complete the operation.

Now a node might have two, four, sixteen or thirty-two slices.

It depends on the resource capacity of that node.

Redshift is a VPC service and so all parts of the system can be managed as you would expect.

So this includes VPC security controls, IAM for permissions, KMS for at rest encryption of the data and CloudWatch can also be used for monitoring of the platform.

Now because of the way Redshift is architected, it has a feature called Enhanced VPC Routing.

By default Redshift uses public routes for traffic when communicating with external services or any AWS services such as S3 when it's loading in data.

Now if you enable Enhanced VPC Routing then traffic is routed based on your VPC network and configuration.

This means it can be controlled by security groups, network access control lists, it can use custom DNS and it will require the use of any VPC gateways that any other type of traffic requires.

So internet gateways, NAT gateways or any other VPC endpoints to reach AWS and external services.

So if you want advanced networking control then you need to enable Enhanced VPC Routing.

Now that's a good one to remember for the exam.

If you do have any customized networking requirements then you do need to enable Enhanced VPC Routing.

I just wanted to stress that again because it really is a critical point to remember for the exam.

Now let's look at the architecture of Redshift visually before we finish with this lesson.

So we start with a VPC and in there is a subnet in one availability zone.

And let's say inside that we create a Redshift cluster.

Now in the cluster there's always a leader node and it's to this leader node that anything outside of the cluster connects to in order to interact with the cluster.

So things like management applications or work benches or visualization.

All of this interacts with the Redshift cluster via the leader node using ODBC or JDBC style connections.

Inside the cluster are the compute nodes and the slices on those nodes as well as the storage attached to each of the slices in the compute nodes.

Because Redshift is located in one availability zone there are a few ways that Redshift attempts to secure your data.

First data is replicated to one additional node when written.

That way the system can cope with localized hardware failure.

Additionally automatic backups happen to S3 by default around every 8 hours or every 5 GB of data written to the cluster.

This happens automatically and has a configurable retention period.

Now these backups occur to S3 and so you now have data stored across availability zones at this point.

So the data with the Redshift cluster at this point is at least resilient against the failure of an availability zone.

Additionally you can create manual snapshots also stored on S3 but you have to manage the retention yourself as an administrator.

You can also configure snapshots to be automatically copied across AWS regions which provides you with global resilience and an effective way to spin up a Redshift cluster in another region if you have a DR event.

In terms of getting data into Redshift you have a few options.

You could load that data from S3 or you could copy data in from products such as DynamoDB.

You can migrate data into Redshift using the database migration service from other data sources and even AWS products such as Kinesis Firehose can stream data into Redshift.

Now for the exam the really important bits to understand are which products can be integrated, how Redshift fits into architectures, so what it can be used for and how to design a Redshift implementation.

So I've covered most of the important architectural points that you'll need for the exam within this lesson.

Now in the next lesson I want to talk in a little bit more detail about Redshift backups.

It won't be a long lesson but I think it will be worthwhile doing because it will have benefits for the exam.

Now that's everything I wanted to cover in this lesson.

Go ahead and complete the lesson and then when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to cover the Elastocache product.

Now this is one which features relatively often in all of the associate AWS exams and fairly often at a professional level.

It's a product that you'll need to understand if you're delivering high performance applications.

It's one of a small number of products which allows your applications to scale to truly high-end levels of performance.

So let's jump in and take a look.

So what is Elastocache?

Well at a high level it's an in-memory database for applications which have high-end performance requirements.

If you think about RDS, that's a managed product which delivers database servers as a service.

Databases generally store data persistently on disk.

Because of this they have a certain level of performance.

No matter how fast the disk is, it's always going to be subject to performance limits.

An in-memory cache holds data in memory which is orders of magnitude faster, both in terms of throughput and latency.

But it's not persistent and so it's used for temporary data.

Elastocache provides two different engines, Redis and Memcache D, and both of them are delivered as a service.

Now in terms of what you'd use the product for, well if you need to cache data for workloads which are read heavy, so read heavy being the key term that you need to remember at this point, or if you have low latency requirements, then using Elastocache is a viable option.

For read heavy workloads, Elastocache can reduce the workloads on a database.

And this is important because databases aren't the best at scaling, especially relational databases.

Now databases are also expensive relative to the data that they store and the performance that they deliver.

So for heavy reads, offloading these to a cache can really help reduce costs.

So it's cost effective.

Remember that for the exam.

Elastocache can also be used as a place to store session data for users of your application, which can help to make your application servers stateless.

Now this is used in most highly available and elastic environments, so those that use load balancers and auto scaling groups.

But for any systems which need to be fault tolerant, where users can't notice if components fail, then generally everything needs to be stateless and so Elastocache can help with this type of architecture.

Now one really important thing to understand for the exam is that using Elastocache means that you need to make application changes.

It's not something that you can just use.

Your application needs to understand a caching architecture.

Your application needs to know to use a cache to check for data.

If data isn't in the cache, then it needs to check the underlying database.

And applications need to be able to write data and understand cache invalidation.

This functionality doesn't come for free and so if you're answering any exam questions which state no application changes, then Elastocache probably won't be a suitable solution.

So let's have a look visually at how some of these architectures work.

Architecturally let's say that you have an application.

Obviously the Categorum application.

And this application is being accessed by a customer.

In this case Bob.

The application uses Aurora as its back-end database engine and it's been adjusted to use Elastocache.

Now the first time that Bob queries the application, the Categorum application will check the cache for any data.

It won't be there though because it's the first time it's been accessed and so this will be a cache miss.

Which means the application will need to go to the database for the data which is slower and more expensive.

Now when it's accessed this data for the first time, the application will write the data it's just queried the database for into the cache.

If Bob queries the same data again, then it will be retrieved directly from Elastocache and no database reads are required.

And this is known as a cache hit.

It will be faster and cheaper because the database won't be used for the query.

Now with this small scale interaction, it's hard to see the immediate architectural benefit of using Elastocache.

But what if there are more users?

What if instead of one Bob, we have many Bob's?

Assuming the patterns of data access are the same or similar, then we'll have a lot more cache hits and a much smaller increase in the number of database reads.

This will allow us to scale our application and accept more customers.

If the application data access patterns of our user base is similar at scale, then it will mean that most of the increase load placed on the application will go directly onto Elastocache.

And we won't have a proportional increase in the number of direct accesses against our database.

And this will allow us to scale the architecture in a much more cost effective way than if everything used direct database access.

So we can scale the application in a much more cost effective way.

We can deliver much higher levels of read workload and we can offer performance improvements at scale.

So this is a caching architecture and this is a very typical architecture that Elastocache will be used for.

Let's take a look at another and this is using the product to help us with session state data for our users.

So let's say again that we're looking at our Categorum application, but now it's running within an auto scaling group with three EC2 instances and a load balancer.

And it's using Aurora for the persistent data layer.

Now again, we have a user of our application, so Bob.

And this application I'm demoing in this part of the lesson is actually the fault tolerant extreme addition of Categorum.

So even when components of the system fail, the application can continue operating without disrupting our user Bob.

Now the way that it does this is to use Elastocache to store session data.

This means that when Bob first connects to any of the application instances, his session data is written by that instance into Elastocache.

And it's kept updated if Bob purchases any limited edition cat prints.

So the first time Bob connects to any of the instances, that instance writes the session data and keeps the session data updated for Bob using Elastocache.

Now if our application at any point needs to deal with the failure of an instance, where previously the session data would be lost and the application functionality disrupted, the Categorum extreme addition can tolerate this.

If this occurs with Categorum extreme addition, then Bob's connection is moved to another instance by the load balancer.

And Bob's experience continues uninterrupted because the session data is loaded by the new instance from Elastocache.

Now this is another common use case for Elastocache, storing user session data externally to application instances, allowing the application to be built in a stateless way, which in turn allows it to go beyond simple high availability and move towards being a fault tolerant application.

Elastocache commonly helps with either read heavy performance improvements or cost reductions or session state storage for users.

But what's also important for the exam is that Elastocache actually provides access to two different engines, Redis and Memcache D.

And it's important that you understand the differences between these two engines at a high level.

So let's look at that next.

Let's look at these differences.

So the differences between Memcache D and Redis.

Both engines offer sub millisecond access to data.

They both support a range of programming languages.

So no matter what your application uses, you can use both of these engines.

But where we start to diverge is on the data structures that each of the products support.

So Memcache D supports simple data structures only.

So strings, whereas Redis supports much more advanced types of data.

So Redis can support lists, sets, sorted sets, hashes, bit arrays, and many more.

So an example could be that an application could use Redis to store data related to a game leaderboard.

And this keeps a list sorted by their rank on that game.

So as well as storing the actual data, Redis can help you by storing the order of this data.

And this can significantly improve the performance of your applications.

Now, another difference is that Redis supports replication of data across multiple availability zones.

So it's highly available by design.

And that can be used to scale reads by using those replicas.

Memcache D doesn't support replication in that way.

You can create multiple nodes which can be used to manually shard your data.

So storing maybe certain usernames in one node and others in another.

But Redis is the one that supports true replication across instances for scalability reasons.

So in the exam, if you face any questions which ask about multi-availability zone or other forms of high availability or resilience, then you should look at selecting Redis as a possible answer.

Now, additionally, from a recoverability perspective, Redis also supports backup and restores, which means that a cache can be restored to a previous state after a failure.

Memcache D doesn't support that.

It doesn't support persistence.

And so if an exam question is asking about recovery of a cache without any impact to the data that's stored in that cache, then you definitely should look at using Redis rather than Memcache D.

Now, Memcache D does have an advantage in that it's multi-threaded by design.

And so it can take better advantage of multi-core CPUs.

It can offer significantly more in terms of performance when using multi-core CPUs.

A notable Redis only feature is transactions.

And this is where you treat multiple operations as one.

So either all of the operations work or non-work at all.

And this can be useful if your application has more strict consistency requirements.

This is one situation where you would select to use Redis versus Memcache D.

Now, both of these engine types can use a range of instance types and instance sizes.

And I've included a link in the lesson description, which gives you an overview of all of the different resources that can be provided to both of these different caching engines.

Now, you don't need to know in detail for the exam, but just be aware architecturally that instance types and sizes which offer larger amounts of memory or faster memory types are obviously going to give you an advantage when it comes to running ElastiCache.

Now, for the exam, you don't need to be aware of all of the different detail.

Just be aware of the types of architectures which would benefit from an in-memory cache.

So anything that has read heavy workloads, where you need to reduce the cost of accessing databases, where you need submilisecond access to data, or where you need to store user session state data in an external way, not using EC2 instances.

So all of those are architectural scenarios where you might want to look at using an in-memory cache.

Now, be aware, it doesn't come for free.

You do need to make application changes.

And so this isn't the type of solution that you can implement if one of your requirements is that you can't make any code changes to your application.

With that being said, though, that's everything that I wanted to cover from a theory perspective in this lesson.

Go ahead, complete the lesson, and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about Amazon Athena.

This product is one of those hidden gems available within AWS which are really valuable as long as you understand the features that it provides.

So let's quickly jump in and explore the architecture.

So what is Athena?

Well, it's a serverless interactive querying service.

Put simply, it means that you can take data stored within S3 and perform ad hoc queries on that data, paying for only the amount of data consumed while running the query and the storage used within S3 to store the original data.

It has no base monthly cost, no per minute or per hour charges, you just pay for the data consumed.

Now what's really special about Athena is how it handles the structured, semi-structured and even unstructured data that it uses.

Athena uses a process called schema on read.

And the way that I want you to imagine this is like a window or a lens through which you see the data in a certain way but where the original data is unchanged.

Your original data stored on S3 is never changed.

It remains on S3 in its original form.

The schema which you define in advance modifies data in flight as it's read through the schema.

So it translates the original unmodified source data into a table-like structure as it's read through the schema.

As you query the data, the original data is read, left unmodified and the translation only happens within the product during the querying process.

I can't stress this enough, the original data is maintained in its unmodified state within S3.

Normally with databases you create tables and you have to load data into those tables.

The data needs to be in the format of the tables or you need to perform ETL processors which stands for extract, transform and load.

With Athena this isn't required.

You define how you want the data to look in the form of a schema and in a non-modifying way data is loaded through this on the fly.

And then any output can be sent to other AWS services.

Now let's look at this visually because it's going to be easier to understand if you see the architecture.

So Athena starts with the source data which is stored on S3 and conceptually this is read only.

It's never modified.

Now the product supports a wide range of data formats and this is growing all of the time.

Some examples include XML, JSON, comma and tab separated values, Avro, Parquet, Ork and even custom application log formats such as Apache and AWS services such as CloudTrail, VPC Flow Logs and more.

So this data on S3 is fixed.

It doesn't get changed and that's probably one of the services most fundamental concepts that you need to understand.

And it's why I've repeated it probably 10 times already in this lesson.

So inside the product you create a schema and in this schema you're essentially defining tables.

These tables define how to get from the format of the original source data to a table like structure.

So unlike a traditional database where a table is the final structure, in Athena you're defining a way to take the original data and present it in a way that you want which allows you to run queries against these tables.

It's almost like a recipe.

You're defining how to convert from ingredients to a final meal.

It's a method to get from the source data to the structure that you want to be able to query.

So these tables within Athena don't actually contain data like a traditional database product.

They contain information, directives on how to convert the source data to be able to query on it.

So this schema is used at the time of querying when data is read and this is why it's called schema on read.

The data is conceptually streamed through the schema while being queried so it can be queried in a relational style way using normal SQL like queries.

And the output can be displayed on the console, saved or output to other AWS tools.

And all the time for this whole process there's no base or constant cost.

You just pay for the amount of data consumed by the query and you can even optimize the original data set to reduce the amount of data that has to be used for individual queries.

The key thing to understand about Athena going into the exam is that it has no infrastructure.

You don't need to think about setting up any database infrastructure in advance.

You don't need to think about data manipulation in advance and you don't need to load data in advance.

Keep those things in mind when going into the exam.

They will help inform you when Athena is the right choice and when it's not.

So Athena is great in situations where loading or transforming of data isn't desired.

Where you have data already on S3 in a source or raw format and you need to query it without doing any loading or transformation.

In the demo lesson which is coming up next you'll see an example of using a large data set, the open street map data.

If you needed to load and transform that prior to use it would massively reduce its utility.

A benefit of Athena is how you don't need to do any loading or transformation of data in advance.

And this makes it ideal for ad hoc or occasional queries of data in S3.

Why?

Well because you don't need any servers running in advance and you don't need to think in advance about loading or transformation.

You have a business need and immediately run a query.

Athena is also useful if you're a cost-conscious business.

It's great because it's servilous.

You pay for any data read as part of a query.

There are no base costs and no upfront costs.

Again, think ad hoc, sporadic and cost effective.

Athena is also the preferred solution especially in the exam for any queries which involve AWS service logs because it has native support of VPC flow logs, cloud trail logs, elastic load balancer logs, cost reports and much more.

And it can also query data from the Glue Data Catalog and supports web server logs.

And again, these are keywords to look for in the exam.

A newer feature of Athena is called Athena Federated Query.

Now be really careful with this one because I don't want you being confused.

For most situations if you see SQL mentioned or no SQL mentioned or any specific database product then the answer to that question is likely not to be Athena.

But Athena now has the capability to query non-S3 data sources.

Athena uses data source connectors that run on AWS Lambda to perform federated queries.

So a data source connector is basically a piece of code that can translate between a target data source which isn't S3 and Athena.

So you can think of a connector as almost like an extension to Athena's querying engine.

So you've got pre-built connectors which exist for data sources like cloud watch logs, DynamoDB, DocumentDB, Amazon RDS and even other JDBC compliant relational data sources such as MySQL, Postgres and many more.

So Athena Federated Query really is a feature which is going to massively improve the utility of the product.

Now that's all of the theory that you need to be aware of for the product as well as some of the key use cases that you might see in the exam.

So at this point go ahead, finish this lesson and then when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about DynamoDB TTL which stands for Time to Live.

It's a pretty simple concept to understand fully but really powerful and it's one that you'll need to understand for the exam and in the real world.

So let's jump in and I'll try to make this one as brief as possible.

Let's say within DynamoDB that we have this table.

I'm only showing three items but let's assume that the table has 3 million.

It has a partition key in blue on the left and a sort key in green next to that.

And in addition to that it has three attributes, yellow in the middle of the table, then red and then pink.

Now for that table there are going to be many partitions which support the operations on that table and here I'm just showing two.

One partition which stores any items which use the dark blue partition key value and another one which stores currently the two items with a light blue partition key value.

Now the TTL feature lets you define a time stamp which allows items in a table to be automatically deleted at a certain point in time.

You're defining when an item is no longer required so per item you specify a date and a time and at that point the item is marked as expired and then deleted.

Now let's step through how this works.

First on a table you need to enable TTL and pick an attribute to be used for the TTL processors.

The attribute should then contain a number.

It's a value in seconds, the seconds since the epoch which is the first of January 1970.

So if you want an item to expire in one week's time you need to find out how many seconds since that date one week from now is and then put that into the attribute that you pick for TTL.

And you should do that on all items which you want to be affected by the TTL processors.

So when you configure TTL on a table what it actually does is it configures automated processors which run on every partition of that table.

The first process periodically scans all items in a partition comparing the value in the TTL attribute with the current date and time in epoch format.

In effect it's checking if the item should still be viewed as valid.

Where any value in the TTL attribute is less than the current date and time so when the item is no longer valid that item is set to expired.

They aren't deleted yet, they can still be queried and viewed in the table, they're just expired.

Next another automated process which runs periodically and which is independent of the initial one this runs also on all of these partitions.

This time it's looking for any items which are set as expired and when it finds any items these items are deleted from the partitions meaning they're removed from the table.

They're also removed from any indexes and a delete is also added to any streams configured on the table.

Now these delete operations are system events they run in the background and don't cause any performance issues nor do they have a charge.

You can also in addition configure a stream on the actual TTL processors and this creates a 24 hour window of any deletions which are caused by those TTL processors.

And this is useful if you want to have any housekeeping processors where you track the TTL events which occur on tables.

So potentially things like an undelete function or some kind of table auditing.

So this is key to understand delete events are placed into a normal table stream along with any creates or modifies but in addition you can create a dedicated stream which is linked to the TTL processors.

So you can get a 24 hour rolling window of just those deletes.

So TTL in summary allows you to define a per item time stamp which determines when an item is no longer needed.

It's useful if you store items which lose relevance after a specific time.

For example removing user or sensor data after one year of inactivity in an application or retaining sensitive data for a certain amount of time.

Maybe you have regulatory or contractual obligations which mean you need to retain data for say one year three years or five years.

Then you can configure those within a TTL attribute and DynamoDB will automatically remove items once they're no longer relevant.

Now that's everything I wanted to cover in this lesson so go ahead and complete the lesson and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about the DynamoDB accelerator known as DAX and DAX is an in-memory cache for DynamoDB which substantially improves performance but unlike other caches it's directly integrated with the product itself so it's really easy for an architect to implement without lots of additional planning work so let's jump in and take a look.

Now before I focus on DAX specifically I want to spend a few minutes contrasting how an example flow works using DAX versus a traditional in-memory cache so let's assume that we have an application which uses DynamoDB for its persistent data storage so on the top we've got the traditional in-memory cache and on the bottom we've got DAX so the flow using the generic in-memory cache goes something like this first the application needs to access some data and so it checks the cache now if the cache doesn't have the data this is known as a cache miss and if this happens the application then loads the data directly from the database once it has the data which takes longer than getting it from the cache directly it updates the cache with the new data and then any subsequent reads from that point forward will directly load the data from the cache which is called a cache hit and this will be faster.

Now the problem with this architecture is the lack of integration between the cache and the database let's contrast this with DAX and when using DAX there's extra software involved and this is installed on the application instance it's the DAX SDK or software development kit and this takes away the admin overhead from the application because now DAX and DynamoDB are one and the same from the applications perspective the application makes a single call requesting the data and this is handled by DAX if DAX has the data if it's a cache hit and the data is returned directly if not then DAX handles the rest so it goes to DynamoDB to retrieve the data it gets the data adds it back into its cache and then returns that data to the client and the benefit of this method is that it's one set of API calls using one software development kit.

DAX is designed for DynamoDB and so it's tightly integrated with it it's much less admin overhead than using a generic cache so by using DAX and by integrating all of the different API calls into one SDK it makes it really easy to implement caching into your application.

So now that we know the difference between DAX and generic caches let's focus now on exactly how the DynamoDB accelerator is architected.

Now DAX operates from within a VPC and it's designed to be deployed into multiple availability zones in that VPC so like many VPC based computers and services you need to deploy it's across availability zones to ensure that it's highly available.

Now DAX is a cluster service where nodes are placed into different availability zones there's a primary node which is the read and write node and this replicates out to other nodes which are the replica nodes and these function as read replicas.

So with this architecture we have an EC2 instance running an application and the DAX SDK and this will communicate with the cluster and at the other side the cluster communicates with DynamoDB.

Now DAX actually maintains two different caches first is the item cache and this caches individual items which are retrieved via the get item or batch get item operations so these operate on single items and you need to specify an items partition key and if present it's sort key so the item cache just holds items which are directly retrieved in this way.

It also has the query cache which stores collection of items retrieved via query or scan operations but crucially it also stores the parameters used in that original query or scan so it links the parameters that are supplied to that operation with the data that's been returned so it means that whole query or scan operations can be rerun and return the same cached data.

Now DAX is accessed architecturally much like RDS every DAX cluster has an endpoint which is used to load balance across the cluster.

If data is retrieved from DAX directly then it's called a cache hit and the results can be returned in microseconds.

You might get a response back typically in say 400 microseconds.

Any cache misses so when DAX has to consult DynamoDB these are generally returned in single digit milliseconds.

Now in writing data to DynamoDB DAX can use write through caching so the data is written into DAX at the same time as being written into the database.

If a cache miss occurs while reading the data is also written to the primary node of the cluster as the data is retrieved and then it's replicated from the primary node to the replica nodes.

So DAX is a really efficient way of interacting with DynamoDB because architecturally it actually abstracts away from DynamoDB you think you're interacting with a single product using a single set of APIs but behind the scenes DAX is handling the process improvement that comes from caching and that's both for read and write operations.

Now before we finish up I just want to step through some important facts and considerations that you'll need for the exam.

So DAX is a cluster you've got the primary node which supports write operations and replicas which support read operations.

Nodes are designed to implement high availability so if you implement multiple nodes and one of the nodes fails for example the primary node it will have an election and fail over to one of the replicas which will become the new primary node.

Now DAX is an in-memory cache and so it allows for much faster read operations and significantly reduced costs.

If you're performing the same set of read operations on the same set of data over and over again then you can achieve substantial performance improvements by implementing DAX and caching those results and in addition because you're not having to constantly go back to DynamoDB time after time for the same data then you generally do achieve significant cost reductions.

In terms of scaling with DAX you can either scale up or scale out so this means using bigger DAX instances or adding additional instances.

So you can scale in both directions up and out.

Now unlike a lot of caches DAX does support write through which means that if you do write some data to DynamoDB you can write it using the DAX SDK.

DAX will handle that data being committed to DynamoDB and also storing that data inside the cache.

Architecturally you do need to know that while DynamoDB is a public AWS service DAX is not it's deployed inside a VPC and so logically any application which is using DAX will also need to be deployed into that VPC.

So DAX is an in-memory cache and it's designed to reduce the response time of read operations by an order of magnitude taking you down from single digit milliseconds using DynamoDB natively all the way through to microseconds if you use DAX.

So architecturally if you're reading the same data over and over again then you need to look at whether you should be using an in-memory cache.

Now choosing between DAX and a generic in-memory cache this comes down to how much admin overhead you want to manage because DAX is integrated with DynamoDB because it supports this single SDK for accessing the cache and DynamoDB both together as an abstract entity it's a lot less workload if you are using DynamoDB on its own to implement DAX rather than a generic cache.

So if you've got read heavy or bursty workloads then DAX provides increased throughput and potentially significant cost reductions.

So if you find yourself having to apply large RCU values onto a table these can get expensive really quickly and you might be better implementing DAX which will get you better performance and remove that additional cost.

So if you find yourself having performance issues during sale periods or have specific tables or items in a table where there are heavy read workloads against that area of data then you can consider using DAX.

If you've got a workload which is very read heavy with the same set of data again and again being read then you can consider using DAX.

If you've got a type of data layout where a certain type is used more frequently than everything else maybe time series then again you can consider using DAX.

If you really care about incredibly low response times again that's another situation where DAX could be advantageous.

Now situations where DAX is not ideal or any applications that require strongly consistent reads.

If your application cannot tolerate eventual consistency then DAX is not going to be suitable.

If you don't have an application that is latency sensitive if you don't need these really low response times again DAX might not be the right solution.

If your application is right heavy and very infrequently uses read operations then again DAX is probably not the right solution.

So generally if you see any questions in the exam which talk about a caching requirement with DynamoDB then you should by default assume it is DAX and only move away from that assumption if you see significant evidence to suggest something else.

With that being said that is everything I wanted to cover inside this lesson so go ahead complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about DynamoDB global tables, something which will form part of your toolkit as a solutions architect for any global database deployments.

Now this lesson will be entirely based around architectural theory so let's jump in and get started.

Global tables aren't actually that complex, they provide multi-master replication meaning no single table is viewed as the master and others replicas instead all tables are the same.

It's global and allows for read and write replication between all tables that are part of a global table.

To implement global tables you create tables in multiple AWS regions and then on one of the tables it doesn't matter which you configure the links between all of the tables.

This creates a global table and sets DynamoDB to configure replication between all of the table replicas.

So tables become table replicas of a global table.

So think of a global table as an entity by itself and supporting that global table are individual DynamoDB tables in different AWS regions configured for multi-master replication.

Now between the tables DynamoDB utilizes last right-of-wins for conflict resolution because it's simple and generally it generates entirely predictable outcomes.

So in the event that you've got the same piece of data being written to two different tables at around the same time then DynamoDB will pick the most recent right and it will replicate that to all of the other replica tables that are part of the global table.

So whichever is the most recent will overwrite everything else.

Now because it's multi-master it means that you can read and write to any region and the updates are replicated to all other regions generally within a second.

Now it's really fast and that matters because it means that it can be used for more demanding applications.

In terms of consistency you can perform strongly consistent reads in the same region as data is written to but for anything else it's always eventual consistency.

The replication between tables is asynchronous and so if you have a global application it needs to be able to tolerate eventual consistency.

If you have a global table and one of the replica tables is in the US and one of them is in London if you're writing to one and reading from the other it will always be eventual consistency and so your application needs to take that into account.

If it can't cope with eventual consistency then global tables are going to be problematic.

Now to use global tables you first need to select the AWS regions which will be part of that global table.

For example we could use one of the US regions AP Southeast 2 in Australia and the London region.

Then in each of those regions you'd create a Dynamo DB table.

You would select one of the tables it doesn't matter which but let's use the US in this example and from that table we would add all of the tables into the global table configuration and this would establish the multi master replication between all three tables.

So all three tables can support reads and writes.

Now specifically for the exam you don't really need to be aware of the implementation details.

For the exam the architecture is what matters so be aware that replication is generally sub second.

This depends somewhat on the load on each of the different regions but generally it does occur within a second.

Now globally as I mentioned moments ago it's eventually consistent but you can do eventual or strongly consistent reads as long as it's in the same region.

The replication is multi master which means that all regions can be used for both read and write operations and finally in terms of what this architecture supports well if you want to implement a globally highly available application or you want to improve the global data performance of an application or you want to add global disaster recovery or business continuity capability to your application then global tables can support all of those requirements.

It's a feature which is really simple to use but it's highly effective and as long as you're aware that the conflict resolution is last right or wins and your application is able to tolerate that then you can use the feature to support a global data layer for your application it's a really good feature to implement as long as your application can tolerate all of the requirements.

So at this point go ahead complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I'm going to be talking about DynamoDB, Streams and Triggers.

Both of these are really powerful features which let you implement some powerful and cost-effective architectures using DynamoDB within AWS.

Now we've got a fair bit to cover so let's jump in and get started.

A DynamoDB Stream is a time-ordered list of changes to items inside a DynamoDB table.

So every time a change occurs to an item in a table, a change is recorded chronologically within a DynamoDB stream.

And a stream is actually a 24-hour rolling window of these changes.

Behind the scenes it actually uses Kinesis Streams which you've covered earlier in the course.

Now you need to enable streams on a per-table basis and when you do enable streams on a table, any inserts of items, updates on items or item deletions are recorded within the stream.

Now you can configure the stream with different views and this is an option on a per-stream basis and the view setting influences exactly what information is added to the stream every time an item change occurs.

Now let's look at this visually because it's much easier to understand.

Let's say that we have a table within DynamoDB and this table has one item, the top item.

And let's say that that item is updated, changed to be the bottom item.

So this represents one item which has changed and the way that it's changed is by having its fourth attribute removed, the one in dark orange.

Now if we've enabled streams on this table then any inserts, updates or deletes are recorded including the one update that I've just discussed.

There are four view types that a stream can be configured with.

It can be configured with keys only, new image, old image and new and old images.

So that's four different options for different view types that a stream can be configured with.

Now given this change to the table on the top left so the removal of its fourth attribute, the one in dark orange, these different view types have the following differences.

So with keys only as the name suggests, the stream will only record the partition key and optionally any applicable sort key value for the item which has changed.

It would be up to whatever is interrogating the stream to determine exactly what has changed and that would probably require a database read.

So in this example where we've removed the fourth attribute, the one in dark orange, the only information that we get using keys only is that the partition key is blue and the sort key is green.

We don't see exactly how this item has been manipulated.

The second view type is new image and this actually stores the entire item with the state as it was after the change.

So if you wanted to configure some sort of business process which operated on the new updated data, then you would configure this view type.

So this view type shows the state of the item after the change.

So after the removal of the fourth dark orange attribute.

If you wanted to know exactly what had changed, then you couldn't determine that using this view type.

Now if you wanted to know what changed, then one option would be to use the old image type.

That way you have a copy of the data as it was before the change and you could check the state of the database, specifically the item in this table, to see exactly what updates had occurred on that data.

So by comparing the old image item inside the stream to the item in the database table, you could calculate exactly what had changed.

Another option to be able to determine exactly what has changed is the final view type which is new and old images.

So if you have a business process which needs complete visibility of the change, so before and after and have this visibility independent of the actual table, then you can select new and old images.

And with this view type, the actual entry in the stream stores both the pre change and the post change state of that item.

And that way you can determine the difference without having to consult the database table itself.

Now it's worth noting that all of these types of views work as well with inserted or deleted items.

But in some cases, the pre or post change state might be empty.

So if you delete an item and you're using the new and old images, then obviously your new state will be completely blank.

And if you're inserting an item and use that same view type, then your old state will be blank and your new state will contain the data that you've inserted.

Now streams are powerful in isolation, but where the real power comes from is that streams are actually the foundation for an architecture called database triggers.

So these allow for actions to take place in the event of a change in data.

So this is an event driven architecture that can respond to any data changes in a table.

Now databases like Oracle have supported these for years, but with DynamoDB, they can be implemented in a serverless way.

Now the architectures of triggers simply put is that an item change inside a database table generates an event.

That event contains the data that's changed and the exact data depends on the view type.

When an item is added to a stream, so when an event is generated, then an action is taken using that data.

And the way that this action is implemented within AWS is to combine the capabilities of streams and Lambda.

So you can use streams and Lambda so that a Lambda function is invoked whenever changes occur to a DynamoDB table.

And so you'll use Lambda to perform some type of compute operation in response to a data change that causes a stream event.

And the Lambda function invocation is actually in this architecture, the trigger.

So it's the compute action that occurs based on data change.

So streams and triggers are actually a really powerful architecture.

You might see them used in reporting or analytics scenarios.

If you want to generate a report in the event of a change of a certain item in a database, maybe stock level changes, or if you want to report on popular items in a stock database, then potentially you can use streams and triggers.

They're also really useful for things like data aggregation.

So if stock levels are being manipulated or if a voting app is recording votes and you want to perform some kind of aggregation or tally of all those votes, then potentially you can use streams and triggers.

You can also use it for messaging or notifications.

For example, if you have a messaging application which allows you to create a group chat, you might use a DynamoDB table to store the chat items that are added to that group.

And you could use streams and triggers to send push notifications to all members of that group chat.

So rather than having to poll databases which consumes compute resources even if nothing happens, if you're using streams and triggers, you can respond to an event as it happens and only consume the minimum amount of compute required to perform that action.

Streams and triggers are really used for lots of different things inside architectures, but for the associate level solutions architect exam, you just need to know that you use streams and lambda together to implement a trigger architecture for DynamoDB.

So visually this is how triggers are implemented.

We've got a table, an item change occurs within a table which has streams enabled.

So a stream event is placed onto a DynamoDB stream.

We've selected to use the new and old images type.

So we've got both the new and the old state of that item.

And based on that, that gets sent as an event to a lambda function and that lambda function can perform some compute based on the pre and post change states of that item.

So it's not a hugely complicated architecture, but it is really powerful.

So enable streams on a table, configure a lambda function to invoke whenever a change occurs and you've got a really cost effective and powerful serverless implementation of a trigger architecture.

And that's what you'll need to understand for the exam.

Now with that being said, that's everything that I wanted to cover in this lesson. go ahead and complete the video and then when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about DynamoDB indexes and there are two types, local secondary indexes known as LSIs and global secondary indexes known as GSIs.

Now we've got a lot to cover so let's jump in and get started.

Indexes are a way to improve the efficiency of data retrieval operations within DynamoDB.

We've already talked about how query is the most efficient operation within DynamoDB but it suffers from one crucial limitation that it can only work on one partition key value at a time and optionally a single or a range of sort key values.

Indexes are a way that you can provide an alternative view on data that's inside a base table.

By providing an alternative view you can allow the query operation to work in ways that it couldn't otherwise and there are two types of indexes, local secondary indexes and global secondary indexes.

Now local secondary indexes allow you to create a view using a different sort key and global secondary indexes allow a view with a different partition and sort key.

And for both of those indexes when you're creating them you have the ability to choose which attributes from the base table are projected into them and choosing what to project is important because it can massively impact how efficient the indexes are for your queries.

Now first I want to focus on local secondary indexes.

So again it's an alternative view on base table data and local secondary indexes must be created with the base table itself.

So this is critical to know for the exam.

You cannot add local secondary indexes after the base table has been created.

And so while you're creating the base table you can optionally create a number of local secondary indexes and the current maximum is five local secondary indexes per base table.

So just to repeat this again because I want this committed to your memory, local secondary indexes allow for an alternative sort key on the data in the main table.

So it's an alternative sort key but the same partition key and local secondary indexes share the capacity with the main table.

And so they use the same RCU and WCU values if the main table is using provisioned capacity.

Now in terms of the attributes that are projected into a local secondary index the options that you have are to use all of the attributes for the base table.

You can choose keys only or you can use include which lets you specifically pick which attributes from the base table are projected into the index.

Now let's take a look at an example visually so that you can understand how this all fits together from an architecture perspective.

And the example that I want to use is a weather station.

So this table holds data for a number of weather stations and for each weather station there's one record taken each day at the same time.

Now if we want to stick to using the query operation then we're limited to querying a single weather station and for a single weather station to either a single day or a range of days.

But what if we want to interrogate data based on another attribute say for example the sunny day attribute.

So this attribute records whenever the average over a day is classified as a sunny day.

Now because this attribute isn't a key in order to perform any operations on it we couldn't use query because query needs to use a single partition key value and optionally select using the sort key we would need to use the scan operation.

And we know now from an earlier lesson that the scan operation is incredibly inefficient.

An option that we have to fix this problem is while creating this table we can also create an additional local secondary index using the sunny day attribute as the sort key.

So this needs to be created along with the base table so at the same time as you're creating the base table.

But if we choose to do that then for a given weather station we're able to easily limit the data that we retrieve to sunny days because we can use the query operation on a single station ID which is still the partition key but then use the query operation to limit specific values in the sort key in this example picking only sunny days.

Now what's even better is that indexes are known as sparse indexes and this means that only items from the base table which have a value for the attribute that we define as the new sort key are present in the index.

Now this means that if the sunny day attribute is something which is present if it's a sunny day and absent if it's not then the only items in the sunny day local secondary index will be for data which is a sunny day.

So we can in some cases take advantage of the fact that indexes are sparse and we can use a scan operation against this local secondary index knowing that we'll only consume the capacity for data that is relevant towards.

So we could use a scan operation on this local secondary index knowing that any items in this index are by default for sunny days and so we're only going to be consuming capacity that's relevant for sunny day data.

Now this is a lot more complex than you'll need for the solutions architect associate exam.

For the exam all you need to know is that local secondary indexes allow you to create an alternative view on the data that's in a base table by providing an alternative sort key.

They use the same partition key and they can only be created along with the base table.

So now let's look at a different type of index a global secondary index.

Global secondary indexes are different than local indexes they can be created at any time and so they're much more flexible.

There's also a default limit of 20 global secondary indexes per base table so you can have more than them and they let you define a different partition and sort key and they also have their own RCU and WCU capacity values if you're using provisioned capacity for the base table.

And just like with local secondary indexes you have the flexibility to choose exactly what attributes are projected into the index from the base table and you have the same options either choosing all attributes keys only or including specific attributes.

So let's look at another example visually to make it easier to understand.

So we're using another example of the weather station this time we have the pink attribute which represents any items where there's been an alarm at the same point as taking the data.

So an alarm could be something like a system error it could be a bird or other wildlife which has entered the weather station and interfered with the results or anything else that's out of the ordinary.

Now if there's a regular access pattern where you need to query this table for any items which have been impacted by the alarm then you couldn't do this normally using a query you'd need to use a scan operation and filter it based on the alarm attribute and this would mean that you're consuming all the capacity for every single item that is read using the scan operation.

Now an option that we have is to create a global secondary index and this allows us to create an alternative view with a different partition key and sort key.

In this example we've created a global secondary index which uses the alarm attribute as the partition key and the station ID as the sort key and this means that we can use the efficient query operation for any items showing an alarm and optionally specify one station ID or a range of station IDs to limit the data that we receive for alarms for specific weather stations.

So global secondary indexes are super powerful because of this ability to define completely separate partition and sort keys so it truly gives you a way to create an alternative perspective on the data that's in a base table and global secondary indexes are also sparse which means in this example any items which have no alarm attribute would not be included in the index.

For the exam you need to be comfortable with the fact that GSIs allow you to create this different perspective on data with alternative partition and sort keys and for GSIs you can create them after you've created the base table so they don't have that limitation of needing to be created at the same time as the base table which is the case for local secondary indexes.

Now one final thing to keep in mind global secondary indexes are always eventually consistent because the data is replicated from the base table to the index asynchronously and so your application needs to be able to handle eventual consistency.

If you're using a global secondary index you need to be able to cope with eventual consistency because that's the only option that you have.

Now before we finish up the lesson I just want to talk through some local and global secondary index considerations things that you should be aware of for the exam.

So you need to be very careful with what attributes you choose to project into the index.

As you now know when you're working with DynamoDB at any time you're reading or writing data you're actually consuming all the capacity for the size of the entire item and so if you project all of the attributes into an index you're also using all the capacity of those attributes so you need to be aware of the capacity that you're using when you project attributes.

Now the inverse of this is if you don't project a specific attribute and then you require that attribute when you're querying an index that will still work but it's doing something in the back end a fetch of that data which is actually incredibly inefficient.

So you need to plan your indexes in advance and make sure that you project the correct attributes because if you are performing queries on any attributes which are not projected then it gets really expensive.

Now AWS recommend using GSIs as default and only using local secondary indexes when strong consistency is required.

So if you need an index and you're in doubt you should use global secondary indexes because they're a lot more flexible and they can be created after the point of when you've created the base table.

Now from an architectural perspective and something to keep in mind if you do see any exam questions which mention indexes is that indexes are designed when you have data in a base table and remember you're designing the base table with the partition and sort keys for the primary way that you will access this data.

Indexes allow you to create this alternative perspective for any alternative access patterns and so if you have a requirement maybe a different team is looking at the weather station data only looking for alarms maybe it's a security team or a data science team then you can create indexes that allow for these alternative access patterns.

Indexes allow you to keep the data in one place but create these perspectives for different types of queries, different teams or different requirements they can all access the same data just using this different perspective.

So at this point go ahead finish the video and when you're ready I look forward to you joining me in the next.

Welcome back.

This is part two of this lesson.

We're going to continue immediately from the end of part one.

So let's get started.

Now DynamoDB can operate using two different consistency modes for read operations.

It can be eventually consistent or it can be strongly or also known as immediately consistent.

Consistency refers to the process of how when data is updated, so when new data is written to the database and then immediately read, is that read data immediately the same as the recent update, or is it only eventually the same?

Eventual consistency is easier to implement from an underlying infrastructure perspective and it scales better.

Strong consistency is essential in some types of applications or some types of operations, but it's more costly to achieve and it scales less well than eventual consistency.

So let's look visually at exactly how this works.

With DynamoDB, every piece of data is replicated multiple times in separate availability zones, and each one of these points is called a storage node.

Out of these three storage nodes in three different availability zones, one of them is elected as the leader, so the leader storage node.

This is a dynamic process.

So if the leader ever fails, the election will happen again and a new one will be chosen.

So in this case, we've got a single item inside a DynamoDB table that's been replicated across three different storage nodes, one in availability zone A, one in availability zone B, and one in availability zone C.

All three have the same data, the same item with the same five attributes.

Now let's say in this example that Bob decides to update some data, it's this particular DynamoDB item, and he decides to remove the fourth attribute, the dark orange attribute.

So this is the change to the item, the fourth attribute is removed, and when writing this to DynamoDB, the product has a fleet of entities which route connections to the appropriate storage nodes.

Writes are always directed at the leader node, so it's this leader node which will receive the update to this item first.

So on the leader node, the item will be manipulated and the fourth attribute, the one in dark orange, will be removed.

At this point, the leader node is known as consistent.

It has the data on it which you just wrote.

That's why writes are more expensive in terms of capacity units.

Why a write capacity unit is less data than a read capacity unit, because writes always occur on the leader storage node.

And so these can't scale as well as reads.

When the leader node has the new data on it, it immediately starts the process of replication.

This process usually only takes milliseconds, but it does depend on the individual load which has been placed on these storage nodes, and it assumes the lack of any faults.

But let's assume though, for this example, that we have no faults on our storage nodes, and we've replicated this updated item to one additional storage node apart from the leader, so the one in availability zone C.

And right now we freeze time.

So the situation we have is that the leader storage node contains that updated item, as well as the storage node in availability zone C, but the storage node in availability zone A does not have the updated item.

It is not consistent.

So right now with time frozen, let's look at reads.

And there are two types of reads which are possible with DynamoDB, eventually consistent reads, and strongly consistent reads.

Now I mentioned earlier in this lesson how you can actually perform reads cheaper than having one RCU representing four kilobytes of data.

Now the reason for this is that one RCU is actually four kilobytes of data read from DynamoDB every second, but that's using strongly consistent reads.

Eventually consistent reads are actually half the price.

So you can read double the amount of data for the same number of RCUs.

So let's say that Julie on the top right performs an eventually consistent read of the data.

When Julie performs that operation and uses eventual consistency, then DynamoDB directs her at one of the three storage nodes at random.

In most cases, all three storage nodes will have the same data.

And so there's little difference between eventual and strongly consistent reads.

But in this particular edge case, if DynamoDB sent her request at the top storage node, then she would get stale data.

Replication occurs in milliseconds, but with eventual consistency, it isn't always guaranteed that you will get the latest data.

And in exchange, because eventually consistent reads scale better because any of the individual nodes can be used, you actually get a price reduction.

It's 50% of the price for strongly consistent reads.

So you get twice as much reads for each individual read capacity unit.

In most cases, you will notice no difference, but you need to be aware that there is a small potential that with eventually consistent reads, you might be reading older versions of data.

If you access DynamoDB at exactly the wrong time, it is possible that you might get outdated data.

Now, in contrast to strongly consistent read, always uses the leader node.

It's always consistent, but because it mandates the use of one particular storage node, the leader node, it's less scalable, and so it costs the normal amount of RCU to perform.

So that's why eventually consistent reads are less cost than strongly consistent.

But one very important thing to keep in mind is that not every application or access type can tolerate eventual consistency.

You need to pick the correct model.

If you have a stock database where the stock level is important, or if you're performing medical examinations and the data that's being logged into DynamoDB is critical and you always need the most recent version, then you need to use strongly consistent reads.

If your application can tolerate a potential lag and the small chance of outdated data, then you can achieve significant cost savings by using eventually consistent reads.

Now, let's just talk through some calculations of how you can actually determine appropriate values for capacity on a table.

Let's look at a scenario and let's assume that you need to store 10 items in DynamoDB every second.

So you have 10 devices that are logging data into DynamoDB and on average, they store data once every second.

So you've got 10 writes per second.

Now, with any type of scenario, you need to determine a number of really important things.

You need to understand the average size of an item that's being written to DynamoDB, and you need to understand how many items per second will be written.

Now, in some cases, exam questions might try to trick you and say that 60 items will be written per minute, and if you see that type of question, you need to try and calculate how many per second because that's what read and write capacity units use.

Now, once you've got both of those pieces of information, you can calculate the WCU required per item.

So to calculate that, you take your item size, in this example, 2.5K, and you divide it by the size of 1 WCU, which is 1K.

That gives us 2.5, and then we need to round that up to the next highest whole number, which in this case is three.

So we know that for a 2.5K average item size, we're going to consume 3 WCU.

And then we need to understand how many of those occur every second, and so we need to multiply that value by the average number of writes per second.

So we know that we're going to store 10 items per second.

We now know that the WCU required per item is three, and based on that, we can multiply those together to get the required WCU, which in this example is 30.

So it's the same calculation every time.

Work out the size of an individual item right in WCU, multiply that by the number of writes per second, and that will give you the WCU setting that you require on the table.

Remember, a WCU is 1K in size.

Now flipping that round, let's look at reads.

If we have a similar example, and we need to retrieve 10 items per second from our database, and we know that the size of an average item is 2.5K, the first thing we need to do is to calculate the RCU that's required per item.

And we do that by taking the average item size and dividing that by 4KB, and then rounding that up to the next highest whole number.

So in this case, because an RCU allows for four kilobytes of reads, we know that 2.5K is going to fit inside 4K.

So every single read is going to be one RCU.

So we know that it's one RCU per read.

So now we know the number of RCU required per item.

We need to know the number per second, the number of operations per second, which is 10, and we multiply those together.

So to do strongly consistent reads with this example, we would need 10 RCU.

But now you know the concept of eventual consistency.

That is half the cost.

So we can take this RCU value and divide it by two, and that means that to perform eventually consistent reads of 10 items per second with a 2.5K size, we need five RCU set on the table.

So I've provided two really simple examples, and what I'm going to do is include some links in the lesson description, which will give you additional examples with different sizes of items, different writes per second, and it will give you some practice on how to perform these calculations.

And I suggest that you do this as extra work once you've finished all of the content of the course.

The only thing that I need you to understand for now is the size of an RCU, the size of a WCU, and exactly how the consistency model works for DynamoDB.

That's all of the theory that I wanted to cover in this lesson though.

Go ahead, complete the video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about DynamoDB operations, DynamoDB consistency and DynamoDB performance.

It's a lot to cover in one lesson so I'll try to be as efficient as possible but let's jump in and get started.

Now DynamoDB allows you to pick between two different capacity modes when you create a table and with some restriction you are able to switch between these modes even after data has been added and these modes are on demand and provisioned.

On demand is a mode which is designed when you have an unknown or unpredictable level of load on a DynamoDB table or alternatively when you have a massive priority for as little admin overhead as possible.

With on demand you don't have to explicitly set capacity settings it's all handled on your behalf by DynamoDB.

You just pay a price per million read and write units but the price that you pay can be as much as five times the price versus using provisioned capacity so it's actually a trade-off.

You're reducing the admin overhead it allows you to cope with unknown or unpredictable levels of demand but you are paying more for that privilege.

With provisioned capacity you actually set a capacity value for reads and writes on a per-table basis so RCU stands for read capacity units and WCU stands for write capacity units.

Now a critical thing to understand is that every operation on a DynamoDB table consumes at least one unit so one unit of read or write.

Now I've added an asterisk here because there is a way to get cheaper reads but I'll introduce this later in this lesson.

One RCU allows for one read operation of up to four kilobytes on a table every second.

If you perform an operation and it only uses one kilobyte to read an item you still consume one RCU.

It rounds up to at least one RCU.

But one operation can consume more.

An item as you learned earlier in the course can be up to 400 kilobytes in size as a maximum and this would consume 100 RCU to read in one operation.

Now a capacity unit is per second so one RCU lets you read one block of data up to four kilobytes every second.

For writes one write capacity unit is one kilobyte so it's the same logic but one kilobyte instead of four kilobytes.

So you set a certain amount of read and write capacity and that gives you a certain amount of read and write load every second.

As well as that every single table within DynamoDB has a WCU and an RCU burst pool.

It calls 300 seconds of the read and write capacity units set on the table.

So when setting read and write capacity units you're setting for the sustained average.

But try to dip into the burst pool as infrequently as possible because other table modification tasks can use this pool as well.

Relying on it too much is pretty dangerous.

If you ever deplete the pool and have insufficient capacity set on a table then you will receive a provisioned throughput exceeded exception error and you'll be throttled.

And the solution is to wait and retry or increase the capacity settings.

Now there are a number of different types of operations that you can perform on a DynamoDB table and some of the most common ones that are mentioned in the exam are query and scan.

So I want to just talk about exactly how these work at a high level before we move on.

The query operation in DynamoDB is one way that you can retrieve data from the product.

When you're performing a query operation you need to start with the partition key.

You have to pick one partition key value.

Let's look at an example visually because it will be easier to understand.

So this is a simple DynamoDB table.

It stores weather data once per day from a group of weather stations.

So the partition key is the sensor ID and the sort key is the day of the week.

And we have one new table for every week of every year.

And to keep things simple these are the sizes of each item.

So item one is 2.5k, item two is 1.5k, item three is 1k and item four is 1.5k.

Now the query operation can return zero items, one item or multiple items.

But you always have to specify a single value for the partition key.

So you can only ever with this example query for one specific weather station.

So regardless of whether your table uses a simple primary key with just the partition key or whether it uses a composite primary key which uses both the partition key and the sort key with query you always have the option of just querying with a single value for the partition key.

So in this example if we decided to query for all items for weather station ID of one then we would get two items returned.

The item for Monday for weather station ID one and the item for Tuesday for weather station ID one.

Now the first item has a size of 2.5k and the second item has a size of 1.5k.

So both of these together equal 4k and one RCU allows you to query 4k.

So this particular query would use one RCU of capacity.

Now with DynamoDB it's always more efficient to return multiple items in a single operation.

In this example we could actually perform a query where we provide the partition key value of one as well as a specific value for the sort key.

So if we wanted to retrieve both of these items with two separate query operations then we could query for a partition key value of one and a sort key of Monday and a partition key value of one and a sort key of Tuesday as two separate operations.

But because every operation consumes at least one RCU then if we ran two separate queries then the same amount of data would consume two RCU.

So it's always more efficient to pull back as much data as you need in totality in one single operation.

Now if you only want to retrieve one specific item then you can query for one particular value of the partition key and one particular value of the sort key.

In this example we could query for a partition key value of one and a sort key value of Monday.

And this would return one single item, the Monday item for Weather Station ID 1 which has a size of 2.5k.

But because it's a single operation and it's less than 4k it will be rounded up to the next whole RCU value.

So this single item query will cost one RCU and it will retrieve the entire item.

Generally with any operations on DynamoDB you always have to operate on the entire item so reading and writing an entire item.

And so there is an architectural benefit with the platform to minimizing the size of an item as much as possible because if you have to perform queries which operate on single items as a minimum you are going to consume the capacity that that whole item uses.

Now just to restress the important thing about queries is that you have to query for one particular value of the partition key and when you're querying for that one value you can retrieve all of the items with that one value or you can filter that down based on supplying one sort key value or a range of sort key values.

And when you do that using the query operation you're only charged for the capacity of that query operation.

So if you pick a particular subset of sort key values you're only charged for the response from that query operation.

What you can do with the query operation is specify particular attributes that you want to return.

So in this example we might only want to return the yellow attribute and the pink attribute but you are still charged for the entire item.

Anything that you filter is discarded but you are still charged for it.

Now if you want to perform a search across an entire table maybe looking for every single weather station entry which indicates good weather you can't do that with the query operation because a query operation can only ever query it based on one particular partition key value.

If you want to perform more flexible operations you need to use scan.

Scan is the least efficient operation within DynamoDB when you want to get data but it's also the most flexible.

Let's use the same example the weather stations.

The way that scan works is to move through the table item by item.

You can specify any attributes that you want to match.

You can show all items for example where a temperature is between two values or retrieve all of the items across all of the weather stations for a given day.

For example Monday.

But what you need to understand about scan is that it is scanning through the entire table.

So the entire table is consumed.

So while you can use it to get access to any data that you want the consumed capacity is for all of the items that are read.

Even if you filter things even if you return less data than the whole table you consume all of the data that's scanned through.

So scan is super flexible but it's also really expensive from a capacity perspective.

So let's quickly look at an example visually.

Let's say that you want to scan the weather table looking for all items across all different weather stations looking for any entries which indicate a sunny day.

So we're looking for a particular attribute which defines that the yellow column and we want to return all items which have this attribute.

Now we can't use a query operation for this because query as I mentioned on the previous screen only allows us to query for one particular value of the partition key.

And we need to look across all different weather stations so multiple values for that partition key.

So we can't use query but we can use scan.

So in this example we're trying to scan for the sunny day attribute, the attribute in yellow and we're looking through the entire table.

So this attribute isn't a partition key and it isn't a sort key and we can't use query.

But we can use scan.

So scan will step through the entire table.

So all four items.

But because we've specified to the scan operation that we only want items which have this sunny day attribute it doesn't return some items.

It doesn't return the ones with the non sunny days.

But that data is just discarded.

We still consume the entire capacity.

So the scan operation would need to step through every item in this table to determine which ones do have the sunny day attribute and which ones don't.

So in this example we actually consume all of the item capacity in the table.

So that's 5k plus 4k plus 2k plus 3k.

So that's a total of 14k which rounded up to the next highest RCU represents 4rcu of capacity that we've consumed.

Even though we're only actually returning 5k plus 4k of data the remaining items which are not valid the ones which don't have sunny days are simply discarded.

Okay so this is the end of part one of this lesson.

It was getting a little bit on the long side and so I wanted to add a break.

It's an opportunity just to take a rest or grab a coffee.

Part two will be continuing immediately from the end of part one.

So go ahead, complete the video and when you're ready, join me in part two.

Welcome back.

Over the next few lessons, I'm going to be stepping through the architecture, features, and important considerations of the Amazon DynamoDB product.

DynamoDB is a no-sequel, wide-column database as a service product within AWS.

It's the database product which tends to be used for serverless or web-scale traditional applications inside AWS.

And for the exam, it's essential that you understand it fully.

Now we have a lot to cover, so let's jump in and get started.

So DynamoDB is a no-sequel database as a service product.

It's a public service which means it's accessible anywhere with access to the public endpoints of DynamoDB.

And this means the public internet or a VPC with either an internet gateway or a gateway VPC endpoint.

DynamoDB is capable of handling simple key value data or data with a structure like the document DB model.

Now strictly speaking, DynamoDB is a wide-column key value database.

And being a database as a service product means that you have no self-managed servers or infrastructure to worry about.

It's not like RDS or Aurora or Aurora serverless which a database server as a service product.

With DynamoDB, you actually get the database itself delivered as a service.

And this reduces the complexity and admin overhead significantly of providing a data store for your applications.

Now DynamoDB also supports a range of scaling options.

You can take full control and choose provisioned capacity and then either manually control performance or allow the system to adjust performance automatically.

Or you can use on-demand mode which is a true as a service performance model.

So essentially set and forget.

Now DynamoDB is highly resilient either across multiple availability zones in a region or optionally DynamoDB allows for global resilience.

So a table can be configured to be globally resilient but that is an optional extra.

Now within DynamoDB data is replicated across multiple storage nodes by default and so you don't need to explicitly handle it like you do with RDS.

Now DynamoDB is really really fast.

It's backed by SSD and so it provides single digit millisecond access to your data.

It also handles backups.

It allows for point-in-time recovery and any data is encrypted at rest.

It even supports event-driven integration allowing you to generate events and configure actions when data within a DynamoDB table changes.

So now let's talk about tables within DynamoDB.

Tables are actually the base entity inside the DynamoDB product.

If I'm being precise which by now you know that I like doing DynamoDB shouldn't really be described as a database as a service product it's more like a database table as a service product.

A table within DynamoDB is a grouping of items which all share the same primary key.

Items within a table are how you manage your data within that table.

So think of an item like a row in a traditional database product.

A table can have an infinite number of items within it.

There are no limits to the number of items within a table.

Now when you create a table within DynamoDB you have to pick its primary key.

Now a primary key can be one of two types.

It can either be a simple primary key which is just the partition key known as a pk or it can be a composite primary key which is a combination of the partition key and the sort key which is known as the sk.

So every item in the table has to use the same primary key and it has to have a unique value for that primary key.

If the primary key is a composite key then the combination of the two parts the pk and the sk need to be unique in that table.

Now that's actually the only restriction on data that an item has the unique values for the primary key.

Items can have other bits of data aside from the primary key called attributes but every item can be different as long as it has the same primary key then it can have no attributes, all attributes, a mixture of attributes or completely different attributes.

An item itself can be a maximum of 400 kilobytes in size and this includes the primary key, the attribute values and the attribute names all total together.

With DynamoDB you can configure a table with provisioned capacity or on-demand capacity.

Now capacity is a pretty odd term.

When you think about capacity you probably think about space but in DynamoDB capacity means speed adding capacity means adding more speed more performance.

Now if you choose to use the on-demand capacity model it means you don't have to worry about it you don't have to set explicit values for capacity on a table you just pay for the operations against the DynamoDB table there's a cost per operation.

If you choose provisioned capacity then it means you need to explicitly set the capacity values on a per table basis and there are two terms that you need to understand.

The first is write capacity units or wcu and the second is read capacity units known as rcu.

One wcu set on a table means that you can write one kilobyte of data per second to that table and one rcu or read capacity unit when set on a table means that you can read four kilobytes per second from that table.

Most operations have a minimum consumption so one read of say 100 bytes will consume one rcu as a minimum and I'll be talking more about capacity control later I'm just introducing the terms for now.

Now let's move on and I want to talk about backups inside DynamoDB.

There are two types of backups available in the product first is on-demand backups and these are similar to how manual rds snapshots function so there are full backup of the table retained until you manually remove that backup.

These on-demand backups can be used to restore data and configuration either to the same region or cross region so if you want to migrate data to another region this is one option you can use a backup to restore a table with or without indexes and you also have the ability to adjust encryption settings as part of the restore.

The key thing to remember is that you're responsible for performing the backup and removing the older backups as needed when they're no longer required but DynamoDB does come with another option and this method is called point-in-time recovery it's something that you need to enable on a table by table basis and it's disabled by default.

When you enable the feature on a table it results in a continuous stream of backups a record of all the changes to a table over a 35 day window and from that 35 day window you can restore to a new table with a one second granularity so when you enable this option you can create a new table by restoring this backup from any one second interval in that entire 35 day window so it's a really powerful feature but you do need to enable it explicitly on a per table basis.

Now before we finish up this lesson there are some considerations that you need to be aware of and some of these are really important for the exam so if you see any questions in the exam which mention no sequel then you should probably preference DynamoDB so if you see no sequel mentioned in the exam and DynamoDB is one of the options to answer that question with unless you've got a strong reason to answer otherwise you should probably default to DynamoDB.

On the other hand if you see any question which mentions relational data or a relational database then the answer is likely to not be DynamoDB.

DynamoDB is not suited to relational data it should not be used to implement any form of relational database system it's simply not designed for that and it doesn't include the required features.

If you see any mention of key value mentioned in any exam questions and DynamoDB is one possible answer then again you should probably default to that answer unless you've got a strong reason not to.

Now access to DynamoDB is via the console the CLI or the API you don't have sequel or the structured query language when using DynamoDB.

If you see any questions or answers which mention SQL or the structured query language then that probably excludes DynamoDB as being a correct answer.

Now the billing for DynamoDB is based around a table it's based on the RCU and WCU values that you set on a table as well as the amount of storage required for that table and any additional features that you enable on that table.

So it's a true on-demand database product there's no infrastructure costs for running DynamoDB no base costs you essentially pay for only the resources that you consume either storage operations or capacity requirements that you specify on a table and in addition you are able to purchase reserved allocations for capacity so if you do know that you have a requirement for long-term capacity on a DynamoDB table then you can purchase reservations so in return for a longer term commitment you get a cheaper rate but with that being said that's everything I wanted to cover in this lesson so go ahead complete this video and then when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about a powerful feature of cloud formation called custom resources.

Now we've got a lot to cover so let's jump in and get started straight away.

The way that cloud formation is architected isn't complicated.

You define logical resources within a template and these define what you want cloud formation to do.

So what infrastructure you want it to create.

Cloud formation uses these logical resources in a template to create a stack and this stack creates physical resources.

If you update the logical resources by updating and reapplying a template then the physical resources are updated.

If you remove a logical resource from the template and then reapply that template to a stack then the physical resources are affected in the same way.

Cloud formation doesn't support everything within AWS.

It can lag behind in terms of products or features of those products and there are some things which it just doesn't support or things it never will support.

Cloud formation custom resources are the answer to anything that you want to do in cloud formation that it doesn't support natively.

Custom resources are a type of logical resource which allow cloud formation to do things it doesn't yet support or doesn't natively support or they allow integration with external systems.

Examples of things that you can do with custom resources might be to populate an S3 bucket with objects when you create it or to delete objects from a bucket when that bucket is being deleted.

Something that will normally error.

If you try to delete a cloud formation stack which contains a bucket with objects within it by default it won't allow you to do that.

It will error.

Another example is that you might want to request configuration information from an external system as part of setting up an EC2 instance.

You can even use custom resources to provision non-AWS resources.

So using custom resources the functionality of cloud formation can be extended much beyond what it can support natively.

Now the architecture of custom resources is simple.

Cloud formation begins the process of creating the custom resource and in doing so it sends data to an endpoint that you define within that custom resource.

This might be a lambda function or it could be an SNS topic.

Whichever one you pick it sends an event to this thing.

Whenever a custom resource is created, updated or deleted then cloud formation sends data to that custom resource.

It sends event data which contains the operation that's happening as well as any property information.

And so the custom resource for example a lambda function is invoked and provided with that information.

Now the compute that's backing that custom resource, let's use the example of a lambda function, can respond to cloud formation letting it know of the success or failure and it can pass back in any data.

Assuming a lambda function which backs a custom resource responds with a success code then everything is assumed to be good.

The custom resource is created.

Any data generated by that lambda function is passed back in to cloud formation and it's made available to anything else within the cloud formation template.

So again two options with how you can back custom resources are lambda or an SNS topic.

Now let's look at this visually because it will help you understand the architecture and then I'll show you a practical example from the AWS console.

So let's consider a scenario where a cloud formation template is used to create a stack which creates an S3 bucket and let's look at this without using a custom resource.

We start with a cloud formation template, it's a simple one which contains a simple S3 bucket logical resource and using this template we create a stack and this creates a logical resource inside this stack and the stack creates the corresponding physical resource which is an S3 bucket.

At this point if you deleted the stack it would delete the logical resource which would delete the physical resource and if the bucket is empty everything would work as expected.

But let's say at this point a human gets involved, Gabby and Gabby makes a manual change by adding additional objects into the bucket.

Now we have a problem because the physical resource is out of sync with cloud formation and we have an even bigger problem because we have a bucket with objects inside it.

If we tried to delete the stack at this point cloud formation would attempt to delete the logical resource which would attempt to delete the physical one but because the bucket contains objects the delete operation would fail.

This is just a limitation of S3 and cloud formation.

It's the type of situation you might hit when you're dealing with any complex architecture or when you need to do something that cloud formation just doesn't support and this is one of the situations which custom resources aims to help with.

Let's look at what capabilities custom resources provide us which might help in this situation.

So we start off with the same basic components.

We've got a cloud formation template which creates a stack.

The template though this time has more resources than just the S3 bucket but first it creates an empty bucket as with the above example.

But in addition it has a custom resource and that custom resource is supported by a lambda function.

Now because this custom resource is backed by a lambda function it means that when the custom resource is being created by the stack the lambda function is invoked or executed and it's passed some data.

This data is event data and this data block contains anything given to the resource as properties.

In this example let's assume that the custom resource is provided with the bucket name of the bucket created by the cloud formation stack so the empty bucket.

Now for the sake of example let's say that we've designed this custom resource of this lambda function to download some new objects into this empty S3 bucket and this now means that we have a bucket with objects inside it.

The same problem that we had with the previous example.

Now in addition to the bucket name being provided to this custom resource the event data also contains details of how the lambda function or how the thing that's backing up the custom resource can respond back to cloud formation and this is called the response URL and so the lambda function because it's completed successfully sends a response back a success response to this response URL and this means the logical resource will create successfully and the stack itself will now move into a create complete status so all is good.

So we've used a custom resource at this point to download some additional objects into that S3 bucket so now we have a cloud formation stack in the create complete status and an S3 bucket with some objects within it.

Now at this point let's say that we have a human come along let's say it's Gabby again and Gabby uploads some additional objects to the S3 bucket and let's say there's some additional cat images so we still have a bucket with objects only now it's more objects than the custom resource added earlier so we've got the objects added by the custom resource and the three additional cat pictures that Gabby has just manually uploaded.

Now let's say that we're going to do a stack delete operation so we select the stack we right-click on it and we select delete stack this starts the process of deleting the stack now the stack has two logical resources and two corresponding physical resources the bucket with objects and the custom resource which is backed by a lambda function.

Now cloud formation in the above example immediately tried to delete the bucket with objects and that's why it failed but in this example because when the custom resource was created it needed the bucket to already have been created it means that cloud formation knows that the custom resource depends on the bucket so there's a dependency and so when you're deleting a stack cloud formation will follow the reverse of that dependency and so that means the custom resource will be deleted before the S3 bucket.

What happens now is that cloud formation starts the deletion of all of the resources contained in the stack but it starts with the custom resource and the way that it does this is to send the message through to the lambda function the message has a similar structure to when the stack was created so it's an event data block and this time it contains the fact that the stack is being deleted and it still contains the name of the S3 bucket.

The lambda function performs whatever actions are configured for a delete operation which in this case is to remove all of the objects from the S3 bucket and once this has been completed once the lambda function has completed all of its operations it will signal back to cloud formation that this was successful and again this will happen by using the response URL that's contained in the event data.

Once the success response has been completed then the stack will delete the custom resource.

Once the custom resource has been deleted there'll be no further dependencies inside the stack and the stack will go ahead and delete the S3 bucket.

This time this will succeed because the S3 bucket is empty and because the deletion of the S3 bucket completes successfully now because it's empty that means the stack itself can be deleted and the process complete successfully so by using a custom resource we can add additional capability to cloud formation.

We can make it download additional objects into an S3 bucket and also have it clean up any objects including those added by a human being outside of cloud formation before the S3 bucket is being deleted and by doing it this way it means we can avoid any stack deletion issues caused by any buckets which contain objects so this is a simple example of how we can extend the functionality of cloud formation by using custom resources.

With that being said that's everything I wanted to cover go ahead and complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to briefly cover CloudFormation change sets.

Now this is a feature which makes it safer to use CloudFormation within a full infrastructure as code environment or when CI/CD processes are being used within your organization.

So let's jump in step through what change sets are and what benefits they provide.

The usual flow that you engage with with CloudFormation goes something like this.

You take a template, use it to create a stack which creates physical resources based on the logical resources in the template.

That's a create stack operation.

Or you delete a stack which deletes the physical resources created by the stack.

Or you can take a newer version of a template, maybe it has additional resources or maybe it's a bug fix.

In either case you take that new template, apply it to an existing stack and this changes existing physical resources and this is known as an update stack operation.

When a stack update occurs, when logical resources are changed which results in changes to physical resources, that change has one of three effects.

We have no interruption and this is where certain changes made to a stack might not impact the operation of the physical resource.

The change is just made and that's it.

Next is some interruption which might mean something like an EC2 instance rebooting.

It's not a damaging event but it can impact service.

And finally certain changes might cause a replacement which creates a new copy of that physical resource and the old one is removed.

This is disruptive and can result in data loss so it's critical to keep this in mind when making changes to existing cloud formation stacks.

Now change sets let you apply a new template to a stack but instead of applying the change immediately it creates a change set which is an overview of the changes to be applied to the stack.

What makes change sets powerful is that you can create many different change sets for a stack so you can preview different changes with different new versions of the template and when you've reviewed the change set or change sets then you can choose to discard them or you can pick one to apply by executing it on the stack which creates the stack update operation and updates the logical and physical resources managed by that stack.

Now visually the architecture looks like this.

Let's use a simple example and don't worry we're going to be stepping through this in the console very shortly.

So this example is a cloud formation template which creates three buckets catpicks dogpicks and memes so we use this to create a stack which creates the three s3 buckets so far so good but now let's say that we didn't actually mean to create the memes bucket we only like animal pictures memes are just no good so we create a new template and we use this to update the stack and because we're not using change sets immediately it deletes one of the s3 buckets the memes bucket the memes bucket has been removed from the template its logical resource because of that is also removed and that means the physical resource that the stack managers is deleted from your AWS account.

Now using change sets we can improve this the starting point is the same we create a stack with the version 1 template but instead of using the version 2 template to update the stack instead we create a change set now this is a distinct thing an object which represents the change between the original stack and the new version of the template we can create one or more of these but when we're satisfied we can execute that change set against the stack and this has the same effect as the top method but we have more control and visibility over the changes especially with larger and more complex templates.

Okay so I'm going to move across now to my console and demo how this works in practice if you really want you can also do this in your own environment have included the cloud formation templates within this lessons folder on the course github repository but for this one I would suggest just watching it's probably not something that's worth the effort of implementing yourself I just want you to be aware of how this looks from the console UI.

Okay so I've switched over to my console and to do this demo lesson I need to be logged in as the I am admin user of the general AWS account so that's the management account of the organization and as always I've got the northern Virginia region selected so I'm going to go ahead and move across to the cloud formation console so I'll type cloud formation in the search box at the top and then click to move to the cloud formation console and just before I apply anything these are the templates which I'm going to be using in this really brief demo lesson so we've got template one and this is a really simple cloud formation template which creates three s3 buckets and then template two which is going to be the one that I update the stack with this only has two s3 buckets so the first template creates cat pics dog pics and memes and then the second template only has cat pics and dog pics so let's move back to the console create stack upload a template file choose file and then depending on the course that you're doing inside the infrastructure as code folder or the cloud formation folder or the deployment folder the name will vary depending on the course you should see a change sets folder and in there we've got these two templates template one and template two select template one and click on open then I'm going to scroll down and click on next and I'm going to call this stack change sets click on next scroll to the bottom next again scroll to the bottom and click on create stack so this is going to create our three s3 buckets we can see that it's doing them in parallel cat pics dog pics and memes that should only take a few seconds there we go one more refresh and it's moved into create complete now that that's in create complete if I click on resources we'll be able to see the three logical resources cat pics dog pics and memes and their corresponding physical resources so the three s3 buckets now there's a change sets tab which if we click on you'll see that there are no current change sets for this stack and we can create a change set from here or we can go to stack actions and then create change set for current stack so that's what we're going to do we're going to create a change set so I'm going to click on create change set this dialogue looks much like the one that you would get if you're just updating a stack without using change sets so I'm going to replace the current template I'm going to upload a template file and I'm going to choose template 2 and then once that's loaded click on next click on next again scroll down next again scroll all the way down and then create the change set and we're going to name the change set so I'm going to call it change sets - version 2 you can call this anything you want but I always find that it's useful to have the stack name at the start and then some kind of version indication and if you wanted to type a description you could do that let's say removing memes and then create the change set initially it will show as create pending I'll hit refresh it will show us create complete and now this is a separate entity in its own right we've created a change set we've not actually updated the original stack so if I go back to stacks we'll still see the change set stack if I click on it and go to resources all three of the logical and physical resources still exist so so whilst we've uploaded this new template version and created a change set we haven't actually done anything with this change set so to use a change set let's go and click on the change sets tab and then open up this change set again we'll see a visual list of all the changes that cloud formation has detected between the version of the template the stacks using and this change set and in this particular case it's telling us that an action of remove is occurring against the logical ID of memes and this physical ID so because we've removed this logical resource from this template it's telling us that the logical resource will be removed along with the corresponding physical resource we can click on this template tab and see an overview of the template that's part of this change set in this case it's the template without the memes logical resource if I click on the JSON changes button you'll see a JSON formatted overview of exactly what's happened between the version of the template the stacks using and the version in this change set it's a list of JSON objects and it's one JSON object per change so in this case there's only one change which is logical resource ID memes and the action is to remove and this is how you can get a really accurate overview of the changes between the version of the template that the stacks using and the version of the template that's contained within this change set so it can form part of a really rigorous change management process within your business now just keep in mind at this point I could have the single change set for this stack or I could have 10 change sets and I can list them all individually I can delete them all individually but if I wanted this change set to be applied against the stack I could do so by clicking on execute so I'm going to do that as soon as I click on execute then it's going to run the update stack operation and at this point the changes would be exactly the same as if you just updated the stacks applied a new template and executed that immediately the template that the stack is using will be changed to the one contained in the change set the logical resource in this case will be removed and the corresponding physical resource will also be removed and the end effect of this will be an updated stack it has an update complete status and if we go ahead and click on resources we can see that this memes bucket has been completely removed from this stack and that's a really simple example of how you can use change sets within cloud formation now at this point that's everything which I wanted to demonstrate in this lesson I'm going to go ahead and click on delete and then delete stack and if you've been following along in your own environment you need to do the same to return the account into the same state as it was at the start of this lesson with that being said though that is everything I wanted to cover so go ahead complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this theory lesson I'm going to cover another cloud formation feature called CFN HUP.

Now we have a decent amount to cover so let's just jump in and get started.

Now as a refresher CFN-init is a helper tool which runs on EC2 instances during bootstrapping.

The tool loads metadata stored within the logical resource in a cloud formation stack and it's a desired state configuration tool which applies a desired state to an EC2 instance based on the metadata of that instance's resource.

But, and this is important, it's only run once.

If you change the cloud formation template and update the stack then CFN-init isn't rerun so the configuration isn't reapplied.

CFN-HUP is an extra tool which you can install and configure on an EC2 instance so you're responsible for installing and configuring it but this can be handled within the same bootstrapping process as any other configuration when the instance is launched.

Now CFN-HUP gets pointed at a logical resource in a stack and it monitors it.

It detects changes in the resources metadata.

This occurs for example when you update the template so when you change the metadata and then perform an update stack operation.

When CFN-HUP detects a change then it can run configurable actions and commonly you might rerun CFN-init to reapply the instance's desired state.

So if you change the metadata, update a stack then CFN-HUP will detect this, rerun CFN-init which will apply that change.

The end effect is that when you use both of these tools together any update stack operations which change the metadata can also update the EC2 instances operating system configuration and this is something which isn't normally the case.

Now visually this is how it looks.

We have a CloudFormation template which is used to create a stack and the stack creates an EC2 instance.

Now the template in this case contains some metadata which is for the EC2 instances logical resource and let's assume that this is used initially to configure the instance.

Maybe to configure WordPress or install some other service or application.

Now this is how CFN-HUP interacts in this type of architecture.

Let's say that we change the template and then when we change the template we perform an update stack operation because we've previously installed CFN-HUP on the instance.

This is periodically checking the metadata for the logical resource for that instance in the stack.

When it detects a change we've configured it to run CFN- init and CFN-init then downloads the new metadata for that instance and applies that new configuration.

It's a simple but super powerful architecture and you'll be getting practical experience of using this in an upcoming demo lesson.

For now I just wanted you to be aware of two things.

Firstly if you update a stack that doesn't automatically rerun any bootstrapping on resources in that stack.

So if you want to change the metadata for a resource to include an additional application install that doesn't automatically get applied.

If you're using normal user data that's only by default executed once when you launch the instance.

If you update a stack and change that it's not automatically reapplied to the instance.

So the way that you need to do this is install and configure CFN-HUP as part of the initial bootstrap process.

Configure it to monitor the logical resource for that instance and then when any changes are made it can then initiate another CFN-init to apply that desired configuration.

So you're going to experience this practically within an upcoming demo lesson.

For now though that's everything so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about a feature of CloudFormation called CloudFormation init.

It's another way that you can provide configuration information to an EC2 instance.

So far you've experienced bootstrapping via the user data and this is an alternative.

Now let's just jump in and get started as we've got a lot to cover.

CloudFormation init is a simple configuration management system.

So far you've used user data to pass scripts into an EC2 instance.

Now this isn't a native CloudFormation feature.

What you're essentially doing is passing in a script through EC2 using the user data feature which is an EC2 feature into the operating system running on the instance where it's executed.

Now CloudFormation init is a native CloudFormation feature.

Configuration directives are stored in a CloudFormation template along with the logical resource it applies to an EC2 instance.

So we have an AWS double colon cloud formation double colon init section of a logical resource.

This is part of an EC2 instance logical resource and it's here where you can specify directives of things that you want to happen on the instance.

The really important distinction that you have to understand is that user data is procedural.

It's a set of commands executed one by one on the instance operating system.

You're essentially telling the instance operating system how to bootstrap itself.

You're giving the instance the how.

How you want things to be done.

CloudFormation init on the other hand this is a desired state system.

You're defining what you want to occur but leaving it up to the system as to how that occurs and that makes it in many different ways much more powerful.

Not least of which because it means that it can be cross platform.

It can work across different flavors of Linux and in some cases on Linux and Windows running on EC2 instances.

Now it's also idempotent meaning if something is already in a certain state running CloudFormation init will leave it in that same state.

If Apache is already installed and your CloudFormation init wants Apache installing then nothing will happen.

If CloudFormation init defines a config file for a service and declares that that service should be started and if both of those things are already true then nothing will happen.

It's much less hassle than having to define within your script's logic as to what should occur if something is already the case.

By using the desired state feature of CloudFormation init it's much easier to design and easier to administer because you just need to define the state that you want instances to be in.

Now accessing the CloudFormation init data is done via a helper script called cfn-init which is installed within the EC2 operating systems.

This is executed via user data.

It's pointed at a logical resource name, generally the logical resource for an EC2 instance that it's running on.

It loads the configuration directives and it makes them so.

Now it's probably going to be easier to understand CloudFormation init along with the cfn-init helper tool if we look at it visually.

It all starts with a CloudFormation template.

This one creates an EC2 instance and you'll see this yourself very soon in a demo lesson.

The template has a logical resource within it for an EC2 instance and this has a new special component.

Metadata an AWS double colon CloudFormation double colon init which is where the cfn-init configuration is stored.

Now the cfn-init helper tool is executed from the user data and so like most EC2 logical resources we pass in some user data but note how this user data is very minimal only containing cfn-init which implements the configuration that we define and then cfn-signal which is used to tell CloudFormation when the bootstrapping is complete.

So the template is used to create a stack which creates an EC2 instance.

The cfn-init line in the user data at the bottom is executed by the instance and this should make sense now everything in the user data section is executed when the instance is first launched.

Now if you look at the command for cfn-init you'll notice that it specifies a few variables stack ID and a region.

Remember this instance is being created by CloudFormation.

These variables are actually replaced for the actual values before it ends up within an EC2 instance.

So the region is the actual region that the stack is created in and the stack ID is the actual ID of the stack that we're currently using and these are all passed to the cfn-init helper tool and this allows cfn-init to communicate with the CloudFormation service and receive its configuration and it can do that because the actual values for the region and the stack name these are all passed in via user data by CloudFormation and once the cfn-init helper tool has this data then it can perform the configuration which has been defined within the logical resource.

Now you're going to experience this in a demo which is coming up slightly later in this section but before we do that I want you to focus on the CloudFormation-init section within the EC2 resource on the left so under metadata and then under CloudFormation double colon init.

We're going to come back to config sets specifically but all of those others are known as config keys.

Think of them as containers of configuration directives and each of them contains the same sections.

So we have packages which defines which packages to install, groups which allow us to define directives to control local group management on the instance operating system, users which is where we can define directives for local user management, sources which lets us define archives which can be downloaded and extracted, files which allow us to configure files to create on the local operating system, commands which is where we can specify commands that we want to execute and then finally services which is where we can define services that should be enabled on the operating system.

Now often within CloudFormation-init you'll define one set of config so one config key containing one set of packages, groups, users, sources, files, commands and services but you can also extend this you can define config sets.

You can create all of these different config keys and then pick from that list and bundle them into a config set which defines which config keys to use and in what order.

Now if you look at the CFN-init line in the user data at the bottom of your screen we're using one specific config set called WordPress underscore install and this uses all of these config keys defined on the left so install CFN, software install, configure instance, install WordPress and configure WordPress but we could have others maybe ones which upgrade WordPress or install a completely different application but whatever the configuration we have in the logical resource we use the CFN-init helper tool we specify the stack ID, the particular logical resource, the region and then the config set to use in this case WordPress underscore install.

Now again don't worry if this is a little bit confusing this is just the theory we're going to be doing one more theory lesson about CFN-hub which is another helper tool available within cloud provision and once we've done that theory lesson as well you're going to do a demo lesson which uses both CFN-init and CFN-hub so by the end of that demo lesson you're going to understand how to use both these helper tools both individually and combined to provide a really good bootstrapping and configuration system.

Now that's all of the theory that I wanted to cover in the next lesson as I've just mentioned we're going to be covering the theory of CFN-hub so at this point thanks for watching go ahead complete this lesson and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to discuss another feature of cloud formation which you will need for the exam so let's jump in and get started.

You know by now that when you create a stack cloud formation creates logical resources based on what's contained in the cloud formation template but for each of those logical resources it also creates a corresponding physical resource within AWS.

Now everything which happens inside AWS requires permissions and as you know by now the default permissions within AWS is zero permissions.

Now by default cloud formation uses the permissions of the identity who is creating the stack to create AWS resources.

Examples of this might be an iam user interacting with a console UI or using the command line.

This means that by default you need the permissions to create, update or delete stacks and permissions to create, update or delete any resources for the stacks that you're creating.

So without any other functionality in order to interact with an AWS account using cloud formation you need both permissions to interact with stacks and permissions to interact with AWS resources.

Now for many organizations this is a problem because there are often separate teams who create resources and then others which are allowed to update them or support them.

Cloud formation stack roles is a feature which allows cloud formation to assume a role and via assuming that role gain the permissions required to interact with AWS and create, delete or update resources.

This allows us to use a form of role separation.

One team can create stacks and the permission sets required to implement them and then the identity creating, updating or modifying a stack only needs permissions on that stack and the past role permissions.

So an identity that's interacting with a stack using stack roles no longer needs the permissions to interact directly with the resources themselves and this is really powerful.

It means an admin user could create a stack with an associated role attached and then a non admin user could be given permissions to interact with that stack using that role without ever having to have the permissions to interact with those resources.

Now this is going to be easier to understand visually so let's have a look at that next.

Now we have two main identities in this example scenario.

We have Gabby on the left who is an account administrator for this AWS account and then on the right we have Phil who is a help desk engineer and in the middle we have the AWS account that both of these identities have to interact with.

Now normally if we wanted Phil to create any AWS resources then he would need to be allocated those permissions either by assuming a role himself or by having the permissions attached either to his user directly or via any groups he is a member of.

Now we want Phil to be able to manage the infrastructure via cloud formation only and either of those options would mean Phil could create the resources directly and this we don't want.

Cloud formation can be a great tool that we can use to control the types of things that can be created, modified or deleted by identities which have lesser permissions.

So using stack roles step one would be that Gabby could create an IAM role with the permissions to interact with AWS resources.

This role has the permissions to interact with the resources but crucially Phil can't edit the role and he has no permissions to directly assume the role.

Phil only has permissions to create, update and delete stacks as well as the permissions to pass the role into cloud formation and that's what he does.

He takes a template that Gabby has created earlier and he uses it to create a stack.

While doing so he passes the role that Gabby's created into the stack by selecting it within the console UI.

This means that the role is attached to the stack and it will be used rather than Phil's own permissions when the stack is performing any resource operations on AWS, meaning that Phil doesn't need the permissions to directly manipulate those resources.

When the stack starts creating it can assume the associated stack role and use the permissions that it gets to create all of the resources within the AWS account so it no longer has to rely on the permissions that Phil has directly associated with his IAM user.

Now this is a really simple example but it's an example of role separation.

Gabby as an administrator can create things within the AWS account that Phil can use.

Gabby herself might not even be allowed to use the things that she creates.

Phil on the other hand doesn't have the permissions to create permissions himself he can't edit the role he can only use it and only with cloud formation.

When using it he doesn't need the permissions that Gabby has to create resources he can simply pass this role into cloud formation and then that gives cloud formation the permissions that it requires to interact with resources inside the account.

Now in the exam if you see this type of scenario where an identity needs to use cloud formation to do things that they wouldn't otherwise be allowed to do outside of cloud formation then stack roles is a great solution to allow this.

You can have one IAM admin user provision an IAM role with the permissions required and then give the identity with the reduced access only the rights to pass that role into cloud formation and then the permissions required to interact with stacks in cloud formation and with the combination of those two things the user Phil in this case can perform actions on the AWS account in a safe and controlled way that he otherwise wouldn't have the permissions to do.

Now that's everything that I wanted to cover in this lesson stack roles is not a complicated feature but it's a powerful one and it's one that you'll need to be fully comfortable with for the exam.

With that being said thanks for watching go ahead complete the video and when you're ready I'll look forward to you joining me in the next lesson.

Welcome back and in this lesson I want to talk briefly about a feature of CloudFormation called deletion policies.

Now this is a pretty simple feature to understand but it's one that you'll be using extensively if you're deploying larger production systems into AWS using CloudFormation.

So let's jump in and get started.

So what is a deletion policy?

Well if you delete a logical resource from within a CloudFormation template and then apply that template to an existing stack or if you delete a stack entirely then the default behavior of CloudFormation is to delete the corresponding physical resource.

Now with certain types of resources this can cause data loss.

If you're deleting RDS databases or EC2 instances with attached EBS volumes then deleting these physical resources can actually delete data that lives on those resources.

Now a deletion policy is something that you can define on each resource within a CloudFormation template and depending on the type of resource you're able to specify a certain action which CloudFormation should take when that physical resource is being deleted.

Now the default is that CloudFormation will delete a physical resource when the corresponding logical resource is deleted.

You can also specify retain and that simply means that CloudFormation will not delete the physical resource if the corresponding logical resource is deleted.

So if every logical resource within a CloudFormation template is set to retain then when you delete the stack none of the physical resources will be removed.

They're retained inside the AWS account.

Now for a smaller subset of supported resources you can specify the snapshot option for the deletion policy.

Supported resources include EBS volumes, ElastiCache, Neptune, RDS and Redshift and when you specify the snapshot option for any of these type of resources then before the physical resource is deleted a snapshot of that resource is taken.

So for example if you have an EBS volume defined within a CloudFormation template using the snapshot deletion policy if you delete that logical resource and then reapply it to a stack CloudFormation will delete the physical resource but not before it's taken a snapshot and these snapshots continue past the stack lifetime.

So if you delete a stack and you have snapshots selected then these snapshots are your responsibility to clean up.

You have to clean them up otherwise they'll continue to incur costs because they're essentially storage within AWS and as with any snapshots that comes with an associated cost.

Now one really important aspect of this to understand is that deletion policies only apply to delete style operations.

Now what this means is if you have a logical resource in a template and you remove it and then apply that to a stack or if you delete the stack then that's a deletion operation and in that case then the deletion policy will apply but it's possible that you can subtly change a logical resource in a template and then reapply that to the stack and that will cause the physical resource to be replaced which is essentially the same as a delete and then a recreate.

Now a deletion policy will not apply in this case and if a resource is replaced then any data on that original resource will be lost and so it's important that you understand that this applies only to deletion operations it does not apply to change astrological resources which cause replacement of physical resources.

So this is how it looks visually at the top we create a stack which creates an EC2 instance an EBS volume and an RDS instance on the left this is what happens when we delete that stack so this is the default process that happens with cloud formation and all of the physical resources are removed from the account.

In the middle we have the same scenario but when the retain deletion policy is used in this case all of our three resources the instance the EBS volume and the RDS instance they're all retained they remain untouched in the account after the stack has been deleted.

Now on the right we have the snapshot option and it's important to note that snapshot is not supported for EC2 instances so we can't choose this option for EC2 but for resources which do support it so an EBS volume or an RDS instance the result will be that an EBS snapshot or an RDS snapshot remain after the stack has been deleted so you will be responsible for managing these snapshots if you want to delete them after a certain period of time that will be entirely your responsibility.

Now this is all of the theory that I wanted to talk about in this lesson I'm not going to cover it in any more depth because this is something that you'll get experience of yourself as you're doing the demo lessons in any of my courses for now though I just wanted to introduce you to the theory and I wanted to make sure that you're aware that the snapshot option is not supported on all AWS resource types so that's really critical to understand.

At this point though thanks for watching go ahead and complete this video and then when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I'm going to be covering stack sets.

And stack sets are a feature of CloudFormation which allows you to create, update or delete infrastructure across many regions potentially in many AWS accounts.

At a high level stack sets allow you to use CloudFormation to deploy and manage infrastructure across many accounts and regions in those accounts.

Rather than having to authenticate to each account individually and switch to each region you can let CloudFormation do all of the hard work on your behalf.

Now let's cover the key concepts first before we look at the architecture visually.

First we have stack sets themselves.

Now think of stack sets as containers and these containers are applied to an admin account.

I don't want you to think of the admin account as anything special.

We just refer to it as an admin account when we're talking about stack sets to distinguish the account where a stack set is applied from all of the other accounts where CloudFormation creates resources.

So again stack sets are containers and stack sets contain stack instances.

Now these aren't the same thing as stacks.

You can think of stack instances as references.

So references to actual stacks running in specific regions in specific AWS accounts.

So stack instances reference one particular stack in one particular region in one particular AWS account and a stack set can contain many stack instances.

Now the reason that stack instances are treated separately from stacks is that if a stack fails to create for any reason then the stack instance will remain to keep a record of what happens.

So why is stack failed to create?

So think of a stack instance as a container for an individual stack.

So to summarize stack sets are applied in an admin account.

Stack sets reference many stack instances and stack instances are containers for individual stacks which run in a particular region in a particular account.

Now stack instances and stacks are created within target accounts.

Now target accounts are just normal AWS accounts that we refer to as target accounts because these are the accounts that stack sets target to deploy resources into.

So a stack set architecture consists of a stack set in an admin account referencing stack instances and stacks which are in the target accounts and regions that you choose.

Now each cloud formation stack created by stack sets is just a normal cloud formation stack.

It runs in one region of one account and the way that all of this multi-account multi-region architecture is created on our behalf by stack sets is by using either self- managed roles or service managed roles and service managed roles are where we use cloud formation in conjunction with AWS organizations so all of the roles get created on your behalf by the product behind the scenes.

So you can either use service managed security where everything's handled by the products for you or you can go ahead and manually create roles and then use self-managed roles to get the permissions so that cloud formation stack sets can create infrastructure across many different accounts in many different regions.

Now visually stack sets look like this so using a stack set starts off in an admin account and this is an AWS account and this is the one that's on the left of your screen.

It's in this account that we create our stack set and we'll call this the bucket atron because we need a lot of S3 buckets for storing some cat related images.

Now an important thing to be aware of is that a template used to create a stack set it's just a normal template it's nothing special.

For this example let's assume that we have a very simple template which creates a single S3 bucket.

Using stack sets we also have some other accounts these are called the target accounts in this example we have two different AWS accounts and in each of these accounts we've got two regions it doesn't matter which ones so let's just call them region 1 and region 2.

Now I mentioned on the previous screen that permissions for stack sets either come in the form of self-managed IAM roles or via service managed permissions as part of an AWS organization which the target accounts are members of.

Cloud formation is essentially in either case assuming a role to interact with all of these target accounts.

Now as part of the stack set creation you indicate which organizational units or accounts you want to use as targets you give the regions and then cloud formation begins interacting with those accounts it uses roles for permissions.

Now what it's doing is creating stack instances within each region that you pick within each target account that you select as part of creating the stack set.

Stack instances remember are just containers they're things that record what happens in each stack that's created by stack sets in each region in each account that you select.

So once we're at this point once we've created the stack set once cloud formation has used the IAM roles to integrate with each of the target accounts that you've selected and once the stack instances have been created then the stacks themselves are actually created.

One per region in each target account that you've selected as part of creating at the stack set and this stack creation process in turn creates the resources which are defined within the template.

Now with this example without using stack sets we have two regions and two accounts so this makes a total of four stacks which would need to be created for the desired infrastructure.

Now what if instead of two regions we used them all and instead of two accounts maybe we have 50.

Then the effort reduction provided by stack sets starts to become a little bit more obvious.

So what kind of things can stack sets be used for?

Let's look at that next with some other key concepts that you need to be aware of for both the exam and real-world usage.

Now there are a few terms that you need to be aware of.

The first is concurrent accounts so this is an option that you can set when creating a stack set and this defines how many individual AWS accounts can be used at the same time.

So if you're deploying a stack set which is deploying resources into say 10 different accounts and you define a concurrent account value of two then only two accounts can be deployed into at any one time which means that over 10 accounts you'll be doing five sets of two.

So the more concurrent accounts that you set in theory the faster the resources will be deployed as part of a stack set.

We've also got the term failure tolerance and failure tolerance is the amount of individual deployments which can fail before the stack set itself is viewed as failed.

So you need to decide this value carefully especially for larger infrastructure deployment and management.

Next we've got the term retain stacks so what you're able to do is remove stack instances from a stack set and by default it will delete any of the stacks that are in the target accounts but you can set it so that you can remove stack instances from different AWS accounts and different OUs and different regions and it will retain any of the cloud formation stacks within those regions within those accounts.

So by default it will delete the actual stacks but you can set it to retain them.

Now the types of scenarios you might use stack sets for might include enabling AWS config across a large range of accounts.

You might want to use stack sets to create AWS config rules for things like multi-factor authentication, elastic IPs or EBS encryption or you might want to use stack sets to create IAM roles that are used for cross account access at scale so instead of having to create them in individual accounts one by one you can define a cloud formation template to create an IAM role and then deploy it as part of a stack set.

Okay now at this point that's everything that I wanted to cover from a theory perspective in this lesson.

Immediately following this lesson is a demo where you're going to get the chance to experience stack sets within your own environment but for now that's everything that I wanted to cover so go ahead and complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to continue on from the last lesson where I stepped through nested stacks only this time I want to talk about cross stack references which are similar but used in a very different set of architectural scenarios.

So let's jump in and get started.

In the last lesson I talked about this architecture.

I talked about how nested stacks could be used to get past the cloud formation resources per stack limit.

I talked about how if you wanted to reuse templates for modular parts of architecture, for example creating a standard VPC template once and then reusing it, then nested stacks were ideal.

And I talked about how if you wanted to simplify the process of creating large infrastructure using cloud formation you could do it by using nested stacks because a single root stack could create any nested stacks which it needed.

Now the limitation of nested stacks was that if you reused a particular template, say the VPC template, you would only reuse the code not the actual VPC that it creates.

If you implemented 10 root stacks each of which was identical then you'd have 10 application stacks, 10 active directory stacks and 10 VPC stacks which included 10 VPCs.

Now in some cases we want to consume a shared component, for example the VPC, for lots of different implementations.

The problem is the isolation which is a design feature of cloud formation.

Let's say that you have a stack which is a well-structured and secured VPC and you want this to be a shared VPC usable by other application stacks in the same region and the same account.

The issue is that stacks are by design self-contained and isolated.

There's a logical boundary around each stack which means that things in one stack can't be by default referenced in another.

In this example if we were deploying EC2 instances into AppStack 1 and AppStack 2 they couldn't natively reference the subnets created by the shared VPC stack.

Now you could manually add the VPC ID and the subnet IDs into AppStack 1 and AppStack 2 but that means they're static parameters, they're not references and this is where cross stack references come in handy if you want one stack to be able to reference the resources created in another in order to reuse those actual resources.

Now to understand the benefit of cross stack references first understand that because of the isolation of stacks normally the outputs of stacks are only visible from the user interface or the command line.

You can't use the built-in ref function of cloud formation to reference anything from one stack in another.

The exception to this as I detailed in the previous lesson is that root stacks can reference the outputs of nested stacks but that architecture as you also learned means that the stacks are linked in terms of their lifecycle.

Sometimes like when you want to create a shared VPC architecture you actually want a situation where a VPC might have a long running lifecycle and applications which use that specific VPC they might have a short lifecycle so you don't want to define all of those as part of the same nested stack.

An example of this imagine you work for a software development company each time a new version of your application is committed to a github repository you want to use cloud formation to create a VPC run the application in the VPC and operate a set of tests before tearing it all down.

Now you could create an isolated VPC each time using a nested stack architecture but if you wanted to save costs you could use the same shared VPC the same set of NAT gateways and the same set of subnets to do this outputs of a template can be exported.

An export is defined within an output of a stack it takes that output and adds it under an exported name to a list of exports in one region of your account so the export name has to be unique.

So for a shared VPC design some examples of what you might choose to export might be VPC ID, subnet IDs, side arrangers, security group IDs anything which you could expect to use elsewhere so external to that shared VPC stack.

To repeat though the export name needs to be unique inside one region of your account.

To use the export inside another stack instead of using the ref function which is how you reference other resources in the same stack you use the import value function you provide import value with the export name and it returns the value exported in that other stack so that's how you can use exports from one stack in another.

So let's have a look visually at how this works.

Architecturally we start with a single AWS account running in one particular region in this example US East 1.

Inside this region we have a VPC stack and we want this to be a shared services VPC which can be used by other stacks.

Step one is inside that stack make sure that anything we want to use is added as an output for example the VPC ID.

So this is an example of the output section for this particular stack we've got shared VPC ID as an output and then we use the reference function to reference the actual VPC logical resource that's created inside this stack.

Now this means this output will be visible from the command line or the console UI but to use this value in any other stack in this region of the account we need to use the export directive to export that value to a list within one region of your account and this is the exports list and this operates per region per account so this is only visible inside your account in one specific region.

Every region inside your account has its own list of exports everyone else's accounts and all of the regions in those accounts each of those has their own dedicated list of exports so within the exports list any of the exports need to be unique so we can only have one export called shared VPC ID within that region of that account.

Now once a value is in the exports list it can be referenced in other stacks using the import value function this function replaces the ref function and remember the ref function is what you can use to reference other logical resources inside a single stack the import value function when used in a stack allows you to reference values which are exported from other stacks and added to the exports list.

Now this only works in the same region as a stack is being applied in cross region or cross account isn't supported for cross stack references so essentially the process is that you need to create an output in one stack export the value for that output into the exports list and then use the import value function to import that exported value into each stack that you want to use it in and that's how you can create shared services by using cross stack references.

Now there are a number of situations where you would choose to use cross stack references for example when you're implementing service oriented architectures i.e. when you need to provide services from one stack to another another example is if you have a churn of short lived applications which all consume from a shared services VPC then you don't want them to be in the same stack or as part of a nested stack if you have things which have different life cycles long versus short then you want to separate them into different stacks and use cross stack references if you want to reuse a stack so reuse the resources created by a stack rather than reusing a template then cross stack references are ideal for the exam I want you to be clear that a template is not a stack or vice versa a template is used to create one or more stacks each stack is unique if you want to reuse a template then you can choose to use nested stacks which allow you to use the same template that you've created once in many distinct architectures a VPC template for example might be used as part of an email system a financial system or for the implementation of hundreds of different isolated client environments that's if you want to reuse a template and each time you reuse that template it creates its own distinct infrastructure but if you want to reuse an actual stack so the resources inside a stack as with this example of a shared VPC then you should use cross stack references rather than nested stacks so cross stack references allow you to reuse actual resources nested stacks allow you to reuse templates they're very different things I hope by this point it makes sense understanding the differences for the exam is essential and if you pick the wrong one in a real-world situation the results can be less than ideal at this point though that's everything that I wanted to cover so go ahead and complete this lesson and when you're ready I look forward to you joining me in the next.

Welcome back.

In the next two lessons I want to cover two features of Cloud Formation.

In this lesson I'm going to be covering Cloud Formation nested stacks and in the lesson following I'll cover cross stack references.

Now we've got a lot to cover so let's jump in and get started.

Most simple projects and deployments which use Cloud Formation will generally utilize a single Cloud Formation stack and a Cloud Formation stack is isolated meaning it contains all of the AWS resources that the project needs.

These might be things such as a VPC, DynamoDB, S3, maybe EC2 and Lambda, SNS, SQS and maybe even a directory service.

Now there's nothing wrong with having a Cloud Formation stack built in this way.

It's isolated, the resources inside it are created together, they're updated together and eventually they're deleted together.

The idea is that all of the resources within a Cloud Formation stack share a life cycle.

Stacks make it simple to package everything up into one collection of resources.

Now designing Cloud Formation in this way where everything's contained in one single stack is fine as long as you don't hit any of the limits that might impact your project.

There are a few things that you need to be aware of.

The first is that there is a limit of 500 resources per stack and for larger deployments this could be a problem.

Another issue with isolated stacks is that you can't easily reuse resources.

If you had a stack like this one which created a VPC it's not practical to reference that VPC in other stacks which might also want to use it.

Stacks are by design isolated by default.

You can use the ref function to reference resources from other resources in the same stack but you can't use this to reference resources in other stacks.

So stacks are isolated.

You have to treat them as self-contained groupings of infrastructure which share the same life cycle.

So you create a stack that creates all of the resources, you update a stack that updates all of the resources and eventually you delete the stack and that deletes all of the resources.

Everything shares the same life cycle.

At the professional level or just for any projects which are complex you'll tend to use a multi-stack architecture.

So you'll implement your project using multiple stacks and there are two ways to architect a multi-stack project.

Nested stacks and cross stack references and choosing between them is what I want you to be fully comfortable with for the exam and in this lesson we're going to be starting by looking at nested stacks.

So let's look at that architecture next.

Nested stacks technically are pretty simple to understand.

You start with one stack which is referred to as the root stack.

In this example this stack is both the root and the parent stack.

A root stack is the stack which gets created first.

So this is the thing that you create either manually through the console UI or the command line or using some form of automation.

So the root stack is the only component of a nested stack which gets created manually by an entity either a human or a software process.

Now a parent stack is the parent of any stacks which it immediately creates.

So complex nested stack structures can actually have multiple levels.

A root stack can create several nested stacks and each of those can in turn create additional nested stacks.

So a parent stack is just a way that we can refer to anything which has its own nested stacks.

So in this case this stack is going to be a root stack and a parent stack.

Now a root stack can have parameters just like a normal stack and also have outputs also just like a normal stack and that's because a root stack is just a normal stack.

There's nothing special about nested stacks.

Inside all stacks you have logical resources and examples of these that you've seen so far include S3 buckets, a virtual private cloud or VPC and maybe even a DynamoDB table.

Now you can also have a cloud formation stack as a logical resource and you define it using the type of AWS double colon cloud formation double colon stack and this is a logical resource just like any other only it creates a stack of its own.

So you have to give the nested stack a URL to the cloud formation template which will be used to create it.

So that template will contain its own resources it's just a normal cloud formation template as I've just mentioned it could even contain its own nested stack.

So in this case HTTPS colon forward slash forward slash some URL dot com forward slash template dot yaml is the URL to a template which will be used to create this nested stack the stack that's called VPC stack.

Now you can also provide nested stack resources with some parameters in this example we're creating a nested stack called VPC stack and if the template for VPC stack had three parameters so param one, param two, param three then we would need to provide values into that stack as it gets created.

For every parameter that the template has for this nested stack we need to provide a value as we create it if not the stack creation process will fail.

So in this particular case the template dot yaml file that's used for VPC stack has three parameters param one, two and three and when we're creating it as a nested stack we need to supply parameters for those values that are used to create that stack.

Now the exception to this is if the VPC stack template had default values for its parameters if it has default values then we wouldn't have to provide those when creating it as a nested stack but it's best practice to populate the VPC stack logical resource with parameters for everything which is parameterized within the template that's used to create that stack.

So in this case we have the root stack it currently has one logical resource VPC stack this creates a nested stack resource and we're passing in these three parameter values.

So when the VPC stack nested stack finishes creating then the logical VPC stack resource within the root stack moves into a create complete status and any outputs of that nested stack are returned to the root stack and these can be referenced using the logical resource name of the nested stack so VPC stack and then dot outputs and then the actual output name of the nested stack.

So you can only reference outputs when using nested stacks you can't directly reference logical resources created in any of the nested stacks you can only reference the outputs that you make visible when creating the nested stack.

Now we might also have other nested stacks contained within the root stack and because these are also logical resources they too would be created but they might have dependencies either ones which cloud formation calculates or one where we use the depends on directive to explicitly inform cloud formation that there is a dependency between different stacks for example we might have an active directory nested stack called AD stack which depends on the VPC stack whether it's a self-managed active directory or one provided by directory service it will need to run from a VPC and so it will depend on that VPC and that VPC is getting created within the VPC stack nested stack.

Now the root stack can take the outputs from one nested stack and give them as parameters to another examples of this might be the VPC ID or the subnet IDs of the resources created inside the VPC stack.

Once the AD stack finishes creating it too might have outputs which are then returned to the root stack then we might create another nested stack perhaps an application stack and this might depend on the AD stack maybe it uses active directory for user authentication and once complete this application stack can also provide its outputs back to the root stack maybe this is a login URL for the application itself as each of the nested stacks finished provisioning the resource in the root stack will be marked as create complete and once all of the logical resources for the nested stacks are complete then the root stack itself will be marked as create complete.

Now there are two really important aspects to nested stacks that you need to understand in order to pick between nested stacks and cross stack references which I'll be talking about in the next lesson.

First by breaking up solutions into modular templates it means that these templates can be reused in this example we have VPC stack which is probably something that can be used again and again for different deployments if you upload the template somewhere then many nested stack architectures can use that template and crucially this is reusing the code or the template for a stack it's not reusing the same stack itself so we're not reusing the VPC that's being created by VPC stack what it means is that we can reuse the template that created VPC stack so by uploading the template for the VPC stack other nested stacks can reuse this same YAML template but if you do reuse the same VPC template in another stack it will create a separate VPC the benefit is the ability to reuse the same templates you're not reusing the same stacks now the AD stack template can also probably be used for different projects which use Active Directory but again every time that this particular template is reused it will create a different Active Directory you're reusing the code not the actual resources this isn't the same when we use cross stack references which I'll be covering in the next lesson because then you're actually reusing resources that are created by a stack when using nested stacks you're reusing the template not the actual stack so you generally use nested stacks when you've created individual building blocks so modular templates and you can reuse each of these templates to form part of a single solution which is life cycle linked so you might be able to reuse the template which creates a VPC on lots of different nested stacks but crucially it would always create a dedicated VPC you would not be using the same VPC in the same stack you would be recreating a new stack and new resources each and every time now nested stacks are generally used when all of the infrastructure that you're creating is forming part of the same solution when it's life cycle linked in this example the application needs Active Directory which needs a VPC it's unlikely that one will exist without the other you aren't going to want to switch out Active Directory for another Active Directory or the VPC for another VPC it's likely that these will all be created together operate together and maybe someday be deleted together now nested stacks do allow for a few main benefits before we finish this lesson I just want to summarize them use nested stacks when you want to overcome the resource limit of using a single stack if you have five stacks together as a nested stack you can have 2,500 resources use nested stacks when you're modularizing your templates that way you can create a VPC template once and use it for many implementations but remember if you use nested stacks and each one of those projects will create its own physical VPC with nested stacks you're only reusing the template so the code you're not reusing resources themselves use nested stacks when you want to make stack installations easier this is because you can apply a root stack and have that root stack automatically orchestrate the application of many nested stacks the one single decision point between using nested stacks and cross stack references is only use nested stacks when everything is life cycle linked when everything in the stack structure needs to be created with each other updated with each other and eventually deleted with each other if you're anticipating needing one part long term but not others then nested stacks are the wrong choice if you imagine needing to use the same actual VPC across multiple implementations then cross stack references are probably better suited and we'll talk about that in the next lesson if you want to make frequent changes to one part of an application and not others then it's probably better to have individual non nested stacks and utilize cross stack references which we'll talk about next okay so that's everything I wanted to cover about nested stacks in the next lesson I'll be comparing this to cross stack references so go ahead and complete this lesson and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about a few related features of CloudFormation and those are weight conditions, creation policies and the CFN signal tool.

So let's jump in and get started straight away.

Before we look at all of those features as a refresher I want to step through what actually happens with the traditional CloudFormation provisioning process and let's assume that we're building an EC2 instance and we're using some user data to bootstrap WordPress.

Well if we do this the process starts with logical resources within the template and the template is used to create a Cloud Formation stack.

Now you know by now that it's the job of the stack to take the logical resources in a template and then create, update or delete physical resources to match them within an AWS account.

So in this case it creates an EC2 instance within an AWS account.

From CloudFormation's perspective in this example it initiates the creation of an EC2 instance so when EC2 reports back that the physical resource has completed provisioning the logical resource changes to create complete and that means everything's good right?

Well the truth is we just don't know.

With simple provisioning when the relevant system EC2 in this case tells CloudFormation that it's finished then CloudFormation has no further access to any other information beyond the fact that EC2 is telling it that that resource has completed its provisioning process.

With more complex resource provisions like this one where bootstrapping goes on beyond when the instance itself is ready then the completion state isn't really available until after the bootstrapping finishes and even then there's no built-in link to communicate back to CloudFormation whether that bootstrapping process was successful or whether it failed.

An EC2 instance will be in a create complete state long before the bootstrapping finishes and so even when it's finished if it fails the resource itself still shows create complete.

Creation policies, weight conditions and CFN signal provide a few ways that we can get around this default limitation and allow systems to provide more detailed signals on completion or not to CloudFormation.

So let's have a look at how this works.

The way that this enhanced signaling is done is via the CFN signal command which is included in the AWS CFN bootstrap package.

The principle is simple enough you configure CloudFormation to hold or pause a resource and I'll talk more about the ways that this is done next but you configure CloudFormation to wait for a certain number of success signals.

You want to make it so that resources such as EC2 instances tell CloudFormation that they're okay.

So in addition to configuring it to wait for a certain number of success signals you also configure a timeout.

This is a value in hours, minutes and seconds within which those signals can be received.

Now the maximum permitted value for this is 12 hours and once configured it means that a logical resource such as an EC2 instance will just wait.

It won't automatically move into a create complete state once the EC2 system says that it's ready.

Instead if the number of success signals that you define is received by CloudFormation within the timeout period then the status of that resource changes into create complete and the stack process continues with the knowledge that the EC2 instance really is finished and ready to go because on the instance you've configured something to explicitly send that signal or signals to CloudFormation.

CFN signal is a utility running on the instance itself actually sending a signal back to the CloudFormation service.

Now if CFN signal communicates a failure signal suggesting that the bootstrapping process didn't complete successfully then the creation of the resource in the stack fails and the stack itself fails.

So that's important to understand CFN signal can send success signals or failure signals and a failure signal explicitly fails the process.

Now another possible outcome of this is the timeout period can be reached without the required number of success signals and in this situation CloudFormation views this as an implicit failure.

The resource being created fails and then logically the stack fails the entire process that it's doing.

Now the actual thing which is being signaled using CFN signal is a logical resource specifically a resource such as EC2 or auto scaling groups which is using a creation policy or a specific type of separate resource called a weight condition resource.

Now AWS suggests that for provisioning EC2 and auto scaling groups you should use a creation policy because it's tied to that specific resource that you're handling but you might have other requirements to signal outside of a specific resource.

For example if you're integrating CloudFormation with an external IT system of some kind in that case you might choose to use a weight condition and next I want to visually step through how both of these work because it will make a lot more sense when you see the architecture visually.

Let's start with the example of an auto scaling group which uses a launch configuration to launch three EC2 instances.

These are within a template and that's used to create a stack.

Because I'm using a creation policy here a few things happen which are different to how CloudFormation normally functions.

First the creation policy here adds a signal requirement and timeout to the stack.

In this case the stack needs three signals and it has a timeout of 15 minutes to receive them.

So the EC2 instances are provisioned but because of the creation policy the auto scaling group doesn't move into a create complete state as normal.

It waits.

It can't complete until the creation policy directive is fulfilled.

The user data for the EC2 instances contains some bootstrapping and then this CFN signal statement at the bottom.

So once the bootstrapping process whatever it is has been completed and let's say that it's installing the Categorum application well the CFN signal tool signals the resource in this case the auto scaling group that it's completed the build.

So this CFN signal that's at the bottom left of your screen this is an actual utility which runs on the EC2 instance as part of the bootstrapping process.

And this causes each instance to signal once and the auto scaling group resource in the stack requires three of these signals within 15 minutes.

If it gets them all and assuming that they're all success signals then the stack moves into a create complete state.

If anything else happens so maybe a timeout happens or maybe one of the three instances has a bug then it will signal a failure and in any of those cases the stack will move into a create failed state.

Creation policies are generally used for EC2 instances or for auto scaling groups and if you do any of the advanced demo lessons in any of my courses you're going to see that I make use of this feature to ensure resources which are being provisioned are actually provisioned correctly before moving on to the next stage.

Now there are situations when you need some additional functionality maybe you want to pass data back to cloud formation or want to put general wait states into your template which can't be passed until a signal is received and that's where wait conditions come in handy.

Wait conditions operate in a similar way to creation policies.

A wait condition is a specific logical resource not something defined in an existing resource.

A wait condition can depend on other resources and other resources can also depend on a wait condition so it can be used as a more general progress gate within a template a point which can't be passed until those signals are received.

A wait condition will not proceed to create complete until it gets its signals or the timeout configured on that wait condition expires.

Now a wait condition relies on a wait handle and a wait handle is another logical resource whose sole job is to generate a pre-signed URL which can be used to send signals to.

It's pre-signed so that whatever using it doesn't need to use any AWS credentials they're included in the pre-signed URL.

So let's say that we have an EC2 instance or external server.

These are responsible for performing a process maybe some final detailed configuration or maybe they assign licensing something which has to happen after a part of the template but before the other part.

So these generate a JSON document which contains some amazing information or some amazing occurrence.

This is just an example it can be as complex or as simple as needed.

This document is passed back as the signal it has a status, a reason, a unique ID and some data.

Now what's awesome about this is that not only does this signal allow resource creation to be paused and then continued when this event has occurred but the data which has passed back can also be accessed elsewhere in the template.

We can use the get at function to query for the data attribute of the wait condition and get access to the details on the signal.

Now this allows a small amount of data exchange and processing between whatever is signaling and the cloud formation stack.

So you can inject specific data about a given event into the JSON document, send this back as a signal and then access this elsewhere in the cloud formation stack and this might be useful for certain things like licensing or to get additional status information about the event from the external system.

And that's wait conditions.

In many ways they're just like creation policies.

They have the same concept.

They allow a specific resource creation to be paused, not allowing progress until signaling is received.

Only wait conditions they're actually a separate resource and can use some more advanced data flow features like I'm demonstrating here.

AWS recommend creation policies for most situations because they're simpler to manage but as you create more complex templates you might well have need to use wait conditions as well and for the exams it's essential that you understand both creation policies and wait conditions which is why I wanted to go into detail on both.

Now that's all of the theory that I wanted to cover about creation policies and wait conditions and these are both things that you're going to get plenty of practical experience of in various demo lessons in all of my courses but I wanted to cover the theory and the architecture so that you can understand them when you come across them in those demos.

For now though thanks for watching go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about a feature of Cloud Formation called Depends On.

And Depends On allows you to establish formal dependencies between resources within Cloud Formation Templates.

Now to explain what this is and why it's required, I'm going to step through a few key points and then we're going to look at it visually.

Now when you use a Cloud Formation template to create a Cloud Formation stack, Cloud Formation tries to be efficient.

And the way that it attempts to be efficient is to do things in parallel.

So when it's creating, updating or deleting resources, it's attempting to do this where possible in parallel.

So for example when using one of my demos, you might notice that many resources inside the template are being created at the same time.

Now while it's doing this, it's trying to determine a dependency order.

So for example, it wants to create the VPC first, then create subnets inside that VPC and then create EC2 instances which run in subnets, which run inside a VPC.

So it tries to determine a dependency order or a dependency tree automatically within the Cloud Formation stack.

Now one of the ways that it does this is by using references or functions.

So if an EC2 instance references a subnet and a subnet references a VPC, then it knows that it needs to create the VPC first, then the subnet and then the EC2 instance.

Now the Depends on feature simply lets you explicitly define any dependencies between resources.

So you can formally define that resources B and C depend on resource A and that means that Cloud Formation will not attempt to provision either of those resources until resource A is in a create complete state.

Now in most cases, this built in dependency mapping will work and you won't encounter any issues.

For example, you might see in many of my demos or advanced demos that if the user data that's defined for an EC2 instance references an Aurora cluster, then it's going to wait until that Aurora cluster has been created before creating the EC2 instance.

So it tends to work in 99% of cases, but Depends on is a really useful way that lets you explicitly define this dependency relationship.

So let's take a look at visually at how that works.

So let's say that we start with two Cloud Formation logical resources in a template.

Let's say a VPC and an Internet Gateway.

Now these are different resources and there's no link between the two.

You can create an Internet Gateway without having a VPC and you can create a VPC without having an Internet Gateway.

So there's no implicit or explicit dependency between either of them.

Neither of them require the other to exist.

But consider this, we have an Internet Gateway attachment.

This attaches an Internet Gateway to a VPC.

So logically it requires both of them.

You can see here in orange at the top that we reference the VPC and in blue at the bottom we reference the Internet Gateway.

This creates implicit dependencies.

The Internet Gateway attachment depends on both the VPC and the Internet Gateway resources.

Now it's implicit because we haven't actually formally stated this dependency.

It's just assumed by Cloud Formation because the Internet Gateway attachment logical resource references the other two.

We can't reference a resource until it's in a create complete state.

So until the VPC and the Internet Gateway resources are fully created, they can't be successfully referenced.

And so the Internet Gateway attachment can't be created because it references those other two resources until those other two resources move into a create complete state.

So this implicit dependency feature works in most cases.

In most cases we can allow Cloud Formation to determine this dependency for us.

But there are some exceptions and one very common one which always seems to impact me when creating demos and one which always tends to feature in exams, hint hint, is this one.

This is that we want to create an elastic IP.

So if you're creating an elastic IP and you want to associate it with a VPC that you're creating in the same template.

So let's say that you're associating it with an EC2 instance running in a subnet inside a VPC.

Then it actually requires an attached Internet Gateway.

Otherwise you'll encounter issues.

You might find that when creating stacks using templates sometimes it works or sometimes it doesn't.

You might find when deleting stacks sometimes it works or sometimes it doesn't.

Without formally declaring this relationship whether you can create an elastic IP depends on the random order that Cloud Formation creates these resources.

So if the elastic IP attempts to create before the Internet Gateway attachment has been created then you will get an error.

If you're deleting a stack and Cloud Formation attempts to delete the Internet Gateway attachment before deleting the elastic IP then you're also going to get an error.

Now what you can do to avoid all of these types of issues is to explicitly define a dependency and this is done using depends on as with this example.

So we use the depends on key value.

For the key we use depends on and for the value we specify the resource which this resource formally depends on.

And so this creates an explicit dependency.

The elastic IP will only be created after the Internet Gateway attachment has been completed and the elastic IP will be deleted before the Internet Gateway attachment is deleted.

So using depends on establishes this formal or explicit dependency which ensures that resources are created, updated and deleted in the correct order.

And this is something that's really essential to understand about Cloud Formation to avoid errors when creating larger templates or when studying an exam with an exam question in this area.

So it's definitely something that you need to understand going into any of the AWS exams and also if you're creating larger Cloud Formation templates.

Now it depends on you can either specify a single resource as with this example or you can specify a list of resources if you want to create multiple dependencies.

So keep that in mind.

And again if you're working through any of my demos or advanced demos it's definitely worth the time to look through the underlying Cloud Formation templates and identify where I've put any depends on statements because that will help you understand exactly how this works for production and for exam questions.

At this point though that's everything I wanted to cover so thanks for watching.

Go ahead complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson I want to cover CloudFormation conditions.

Now these are a useful feature of CloudFormation which allows a stack to react to certain conditions and change infrastructure which is deployed or specific configuration of that infrastructure based on those conditions.

Now it's a simple feature but it provides a lot of flexibility for architects, developers or engineers.

So let's jump in and step through exactly how it works and what features it provides.

So CloudFormation conditions are declared within an optional section of the template, the conditions section.

Now you can define many conditions within the conditions section of the template and the end effect is that each condition is evaluated to be true or false.

And these are processed before logical resources which are defined within a template, a processed by CloudFormation and physical resources are created to mirror those logical resources.

So essentially the conditions section of a template is evaluated first and then based on those conditions any logical resources which use those conditions that influences what physical resources are created and how they're created.

So these conditions use other intrinsic functions so AND = IF NOT AND OR and it uses these intrinsic functions to evaluate one or more things and then the result of those functions determines whether the condition itself is true or false.

Any logical resources within a template can have a condition associated with them and the condition that's associated with them defines whether they're created or not.

So if a condition that's associated with a resource is true then that logical resource is created.

If a condition that's associated with a resource is false that resource is not created.

Now an example is you could have a parameter value on a template which accepted a number.

Let's say 1, 2 or 3 and then we could create three conditions within a template.

Let's say 1AZ, 2AZ or 3AZ and each of these conditions would use intrinsic functions to evaluate whether the parameter value was 1, 2 or 3.

Now we could have many duplicate sets of resources defined within the CloudFormation template and certain of those resources would only be created if 2AZ was true and certain resources would only be created if 3AZ were true.

We could also have conditions which react to the environment type parameter of a template.

So based on whether the template was prod or dev we could control the size of instances created by a CloudFormation stack.

So these are just two relatively common ways that conditions are used within a CloudFormation stack and a CloudFormation template.

Now let's take a look at how this looks visually and I'm going to step through a pretty simple example.

So we have three major component parts.

First we have a template parameter.

In this example, EnvType.

An EnvType can be dev or prod and it represents what the template is being used for.

So development activities or production usage.

Then also within the template we have a condition defined inside the conditions block of the template and this uses the equals intrinsic function to check if the value of the EnvType parameter is prod and if it is then this condition is prod is set to true.

Finally conditions are used within resources of the template.

In this case the word pressed to myEIP and myEIP2 resources they all reference this condition.

And just before anyone provides feedback for anyone with a really keen eye these templates are not complete.

So let's just refer to them as pseudo CloudFormation.

They're cut down to only show what matters for this lesson.

The flow through this architecture would start with our developer Bob who would decide on a value of dev or prod for the EnvType parameter when applying it.

So this would set that parameter value.

Now the template as I've just mentioned has the conditions block which is evaluated first by CloudFormation before even considering the resources.

So this evaluates to true or false.

If the EnvType parameter in the template is prod then this condition evaluates to true otherwise it's false.

Now the next stage is that the processing of the resources within the stack begins when processing the resources for any resources which use the isProd condition they're only created if the condition that they reference is true.

So in this example if the isProd condition is false then only the wordpress resource is created.

Because all of the other three have the condition which if it's false will cause those resources not to be created.

Now if the isProd condition was true then wordpress2 would also be created so we'd have two EC2 instances and each of those would also be allocated with an elastic IP.

So myEIP and myEIP2.

So just to reiterate this if a logical resource does not have a condition then it's created regardless.

If a logical resource does reference a condition then that logical resource is only used so a physical resource is only created if that condition evaluates to true.

If it evaluates to false then no physical resource is created for that corresponding logical resource.

Now when you step through the flow of using these conditions they aren't actually that difficult to understand.

You define a condition, you set it to true or false using one of the intrinsic functions and then you use that condition within resources in the template.

Now conditions can also be nested so you could have an isProd condition.

You could also have another condition such as createS3 bucket and if this was true it would create the S3 bucket.

And then you could have a condition which controls if a bucket policy is applied to the bucket and you could configure it so that that bucket policy would only be applied if a bucket is created and if that stack is a production stack.

So you can nest conditions together and make a condition that evaluates to true only if two other conditions also evaluate to true and this nesting is done by using these intrinsic functions.

Now I'll be showing you lots of different examples of conditions in my demos and advanced demos that you'll find through all of my courses.

So always make a habit as you do the demos to review the cloud formation templates which are used.

So seeing these through practical examples will help improve your understanding.

With that you'll be able to read templates and with more and more practice you'll find that writing them becomes much easier.

At this point though that's everything that I wanted to cover in this theory lesson about cloud formation conditions.

Thanks for watching.

Go ahead and complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this video I want to talk about cloud formation outputs and outputs are optional within a template.

Many don't have them but they're useful in providing status information or showing how to access services which are created by a cloud formation stack.

Now this is going to be a really quick video so let's jump in and get started.

So the output section of a template is entirely optional.

You can implement perfectly valid cloud formation templates without using an output section but if you do decide to include an output section then you can create outputs within that section.

Essentially you can declare values inside this section which will be visible as outputs when using the CLI, they'll be visible as outputs when using the console UI and and this is a really important point they will be accessible from a parent stack when using nesting and these outputs can be exported allowing cross stack references.

Now outputs are not a complex topic and so I don't want to dwell too much on how they work because you'll be getting some practical experience in an upcoming demo video.

Visually though this is how it might look if you're declaring a simple output.

So in this example we're provisioning an EC2 instance which is running WordPress and this is an output within that template.

So what we're doing is defining an output called WordPress URL and then we're defining two key value pairs description and value.

So description is something which is visible from the CLI console UI and is passed back to the parent stack when nested stacks are being used.

So you can always access the description and it's best practice to provide a description which makes this useful to anyone who might not have seen the template.

Now the second part is the value and the value is important.

The value determines exactly what you want to be exposed by the cloud formation stack once the stack is in a create complete state.

So in this case what we're actually doing is creating a value by joining two other things together.

So we're using the join intrinsic function which I've covered in a different video and we're joining the literal string of HTTPS colon forward slash forward slash and the logical resource attribute of DNS name and this is how we can create a URL for accessing the service that's created by this cloud formation template.

So we're using the join function to generate a simple string from two different things from HTTPS colon slash slash which is a literal string and then the attribute of the instance which is created elsewhere in this template.

So the output will be HTTPS colon slash slash and then the DNS name of the instance and this will provide a method for anyone who's implementing this template to be able to access the service.

So that's everything I wanted to cover about cloud formation outputs.

They're not all that complicated you'll be getting some practical experience in using them in an upcoming demo video and when I talk about cross stack references you'll see how we can extend this by exporting a particular output or set of outputs but at this point I want to keep things simple and that's everything that you need to be aware of when it comes to cloud formation outputs.

So go ahead complete this video and when you're ready I look forward to you joining me in the next.

Welcome back and in this video I want to talk about CloudFormation mappings.

And in keeping with the theme from the last few videos, this is also a feature of CloudFormation which makes it easier to design portable templates.

Now this is going to be a fairly brief video so let's jump in and get started.

CloudFormation templates can contain a mappings object.

Remember at a top level a YAML or JSON template is just a collection of top-level key value pairs.

Now resources is one, parameters is one and now I'm introducing mappings as another.

The mappings object can contain many mapping logical resources and each of these maps keys to values allowing information lookup.

So you might use mappings to map the environment for example production to a particular database configuration or a specific SSH key.

Now these mappings can have one level of lookup so you can provide a key and get a value back or they can have top and second level keys.

Now a common example is a mapping of AMI IDs based on the top level key of region and the second level key of architecture.

Mappings use another intrinsic function which I haven't introduced yet called find in map and an example which I'll show next is the common use case which I just talked about using find in map to retrieve a given Amazon machine image ID for a particular region and a particular architecture.

Now at this point the key thing to remember about mappings is that they help you, you guessed it, improve template portability.

They let you store some data which can be used to influence how the template behaves for a given input.

So let's have a look at a simplified example visually on the next screen.

Now this is an example of one mapping which is called region map which is in the mappings part of a cloud formation template and this is an example of the find in map function that you will use to lookup data using the mapping.

Now to use a mapping it's actually pretty simple.

First we have to use this find in map function and we need to specify a number of pieces of information to this find in map function.

The first thing that we need to specify is the name of the mapping that we're going to use in this case region map.

So this allows the intrinsic function find in map to query a particular mapping in the mappings area of the cloud formation template.

Now the next part this is mandatory we always need to provide at least one top level key.

In this case we need to provide an item that we will use to lookup information from the mapping.

Now in this case we're using a pseudo parameter.

A WS double colon region will always resolve to the region that this template is being applied in to create a stack.

So in this case let's assume that it's US - East - 1.

Now the mapping to use and this top level key are the only mandatory parts of find in map and if we only provided these two then it would retrieve the entire object below US - East - 1 on this example.

But in this case we're going to provide a second level key, HVM 64.

And if we provide this as well it will perform a second level of lookup.

Meaning in this case we will retrieve the AMI ID for US - East - 1 using the HVM 64 architecture.

So this is a simple example but it's a fairly common scenario where you use the mappings area of a template to store an AMI lookup table that you can use to retrieve a particular suitable AMI for a given AWS region and a given architecture.

Now you could change this, you could use a particular AMI for a particular region and a particular application or a particular environment type.

But you can perform one or two level lookups using find in map.

Now again in a future video you're going to get the chance to experience this yourself in a demo video but for now I just wanted to introduce the theory, the architecture behind mappings.

So that's everything I wanted to cover in this video so go ahead, complete the video and when you're ready I look forward to you joining me in the next.

Welcome back and in this video I want to cover CloudFormation intrinsic functions.

Up until this point everything that you've defined within a CloudFormation template has either been static or accepted using parameters.

While intrinsic functions allow you to gain access to data at runtime, your template can take actions based on how things are when the template is being used to create a stack and that's really powerful.

In this lesson I want to cover the theory of intrinsic functions but don't worry you'll be getting the chance to use them practically in an upcoming demo video so let's jump in and get started.

Now I want to quickly step through the functions that we're going to be looking at over the remaining videos of this CloudFormation series and then we can look at some of them visually and technically step through how they work.

So first we're going to be looking at the ref and get attribute function or get at and these both allow you to reference a value from one logical resource or parameter in another one.

If you create a VPC in a template and you want to make sure that another resource such as a subnet goes inside that VPC then you can reference the VPC within other logical resources.

Next we've got join and split and these as the name suggests allow you to join strings together or split them up.

An example usage might be is if you create an EC2 instance which is given a public IP version for DNS name then you can use the join function to create a web URL that anyone can use to access that resource.

Next is get azs which can be used to get a list of availability zones for a given AWS region and the select function which allows you to select one element from that list and these two are commonly used together to pick an availability zone from the list of availability zones in one particular region.

Next are a set of conditional logic functions if and equals not and or and these can be used to provision resources based on conditional checks.

So for example if a certain parameter is set to prod then deploy big instances.

If it's dev then deploy smaller ones.

Next is base64 and sub.

Many parts of AWS accept input using base64 encoding.

For example if you're providing EC2 with some user data for automated builds then you need to provide this using base64.

So the base64 function accepts non-encoded text and its outputs base64 encoded text that you can then provide to that resource.

Sub allows you to substitute things within text based on runtime information.

So you might be passing build information into EC2 and you want to provide a value from the template parameters in which case sub can help you do that.

And then next we've got sider which lets you build sider blocks for networking.

It's a way to automatically configure the network ranges subnet used within a cloud formation template.

Now there are others such as import value, find in map and transform and I'll be covering these in dedicated videos later in this series.

Each one of these functions can be used in isolation or used together to implement some pretty advanced logic within templates.

Now let's take a look at how these work visually and technically and once again don't worry you will be getting the chance to use all of these practically in upcoming demo video.

Two of the most common intrinsic functions within cloud formation are ref and get at meaning get attribute.

It's important that you understand how these are used and the differences between the two.

So let's use this as an example.

A template with a logical resource which we're going to use to create a stack and this creates a physical resource in this case a t3.micro EC2 instance.

Now every parameter and logical resource within cloud formation has a main value which it returns so for example the main value returned by an EC2 instance is its physical resource ID.

The main value for a parameter logically enough is its value and the ref function can be used as the name suggests to reference this main value of a parameter or a logical resource.

Now if you look at the cloud formation simplified example at the bottom left you will see next to image ID we're referencing latest AMI ID which is a parameter and that's how we can use parameters with logical resources by referencing them.

We can also use ref with logical resources as I just described so when an EC2 instance is created once it reaches a create complete state then it makes available a range of data.

The primary value its physical ID can be accessed using the ref intrinsic function.

Now there are also secondary values depending on the type of resource that you're deploying and these can be accessed using the get at function.

With this function you provide the logical resource name and the name of an attribute and examples of this free EC2 might be the public IP address or the public DNS name of the instance.

Ref and get at are critical.

They're used in almost all cloud formation templates to access logical resource attributes, template parameters, pseudo parameters and much more.

They'll be the key to evolving the non-portable template that you created in the previous demo video through to being a portable template so it's really important that you understand how both of these work.

Next I want to talk about the get azs function and the select function.

Now these are often used together which is why I've included them on the same example.

Get azs is an environmental awareness function.

Let's say that we're deploying a template into US East 1 and let's assume this region has 6 azs, US East 1A, 1B, 1C, 1D, 1E and 1F.

Now if you wanted to launch an EC2 instance into one of these azs you would need to know its name.

Basically you would need to know a list of names for all of the valid azs in that region and then you would need to pick one.

Remember from the previous demo video we're trying to ensure that our templates are portable so hard coding, availability zone names is a bad practice.

What you can do is use the get azs function and with this you can either explicitly specify a region, you can use the region pseudo parameter or you can leave it blank and then it will use the region currently being used to create the stack.

What it will do is return a list of availability zones within that region.

There is a little nuance here though under normal circumstances it should return a list of all the azs in that region but what it actually does is to return a list of all azs within that region where the default VPC has subnets in that az.

Now normally these are one and the same but if you have a default VPC where you've deleted subnets then the list that you're going to get back is not going to have all available azs.

So if you don't have a default VPC or if you have the default VPC in its form where it does have subnets for all azs in that region then it will return a list of all available azs within that region.

But if you have a badly configured default VPC then you might get some inconsistent results.

But having this dynamic list of availability zones is really powerful because then you can use the select function to select a numbered one from that list.

Now select accepts a list and an index starting at zero which returns the first object in that list.

So it allows you to dynamically refer to azs in the current region without explicitly stating their identifiers which makes templates much more portable.

It's one part of ensuring that templates can be applied to all regions without having issues and it's something that you're going to get experience of very soon in the next demo video.

Now I'm going to start moving through the rest of these much faster because some of these intrinsic functions are much more situational and you're going to get experience of them as we move through this series of videos.

Next we have the join function and the split function.

Now split accepts a single string value and a delimiter pipe in this example and it outputs a list where each object in the list is part of the original split.

So in this example we provide split with a single string, ruffle pipe, truffles pipe, penny pipe, winky and we get as an output a list where each object in that list is one of those cat names which can be referenced individually.

Now join is the reverse of this.

You provide a delimiter and a list of values and the join function joins them together to make a string.

In this case we're creating a web URL for a WordPress EC2 instance by combining https// and the DNS name of the instance and note how that's obtained with the get at function which I've just covered.

Okay so moving on, next we have base64 and sub.

Now this is an example of user data which I'm going to be covering soon.

Essentially it's a script that you provide to instances which allow them to perform auto configuration.

Now this user data needs to be provided using base64 encoded text but as you can see this isn't the case.

It's simply using plain text.

The base64 function accepts normal text and it encodes it and then passes the output which is base64 into an instance which is the format that that instance needs.

So if you're operating with any AWS resources which require base64 then you can use this function, provide the function with some normal text and it will output the base64 encoded text that you need.

Now the substitute or sub function allows you to do replacements on variables.

So for example this is a variable.

This is the instance ID attribute of the instance logical resource.

By putting it in this format so dollar, curly bracket, variable name, close curly bracket the sub function will replace it with the actual runtime value, the instance ID.

Now there are some restrictions.

You can't do self references.

So in this case this user data could only reference the instance ID of another instance.

This example is actually an invalid one which I wanted to show you visually.

The formatting is correct but it actually shows a self reference.

How can we pass in an attribute of a physical instance before the physical instance is created?

So this is not valid but during an upcoming demo video I'm going to be covering how to use these effectively and you're going to get plenty of practical experience of using the sub function within your own cloud formation templates.

Now the format of using things in substitutions is either the left one for a parameter, the middle one for the primary value of a resource and this is like using the ref function and the right is the format for using attributes.

The logical resource name and then the attribute name and again don't worry you're going to get plenty of practical experience of using this in an upcoming demo video.

The last function I want to talk about is actually a really cool feature of cloud formation which makes networking much easier.

So when you're creating VPCs you have to provide a side arrange for the VPC to use.

Inside that side arrange you've historically had to manually assign rangers for the sub nets inside that VPC.

With this function you can use it to reference the side arrange in this example of a VPC.

You can tell it how many sub nets you want to allocate and then finally you can tell it the size of those sub nets and from that it will output a list of side arrangers which you can use within sub nets within a VPC and you can combine this with the select function to allocate those to sub nets individually.

So in both of these examples sub net one and sub net two what we're doing is we're using the side function, we're passing it the side arrange of the VPC, we're telling it we want 16 ranges in total and we're giving it the size for those ranges.

Both of them output a list of possible ranges to use and we're selecting the first one so index zero for sub net one and the second one so index one for sub net two.

And this is an example of how we can assign side arrangers to sub nets in a more automated way.

It assists again in making templates more portable by auto assigning things.

Now it does have its limitations, it's all based on the parent VPC side arrange and it can't allocate or unallocate ranges but luckily I'm going to show you some really cool techniques how you can fix that in later videos of this series.

Now at this point that's everything I wanted to cover, I wanted to quickly go through some common intrinsic functions that you might use while you're creating cloud formation templates.

Now very soon there's going to be another demo video where you're going to get some practical experience to all of the theoretical concepts that I've been talking about in this block of theory videos.

So don't worry we start with a theory, we make sure that you're entirely comfortable with that and then you'll get the opportunity to practice that in a demo video.

Now that's everything that I wanted to cover in this video so as always please go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video I want to talk about template and pseudo parameters, two types of parameters which can be used within CloudFormation templates and which can influence logical resources within those templates.

Now we've got a lot to cover so let's jump in and get started.

Parameters both template and pseudo parameters allow input.

They let external sources provide input into CloudFormation.

For template parameters this means that the human or automated process can provide input via the console, CLI or API when a stack is created or updated.

An example of this might be the size of the instance or the environment that the template is for.

So for example dev, test or prod.

Now parameters are defined inside a template along with the resources and the values for those parameters can be referenced within logical resources also within that template which allows them to influence the physical resources and/or the configuration of those physical resources when a template is used with a stack to provision AWS resources.

For every parameter that you define in a template you can provide configuration for that specific parameter.

You can define defaults for it so if no value is explicitly provided then that default applies.

You can define allowed values so maybe a list of instance types which are valid for the template.

You can define restrictions such as the minimum and maximum length or even allowed patterns.

You can also define the parameter as using no echo which is useful for passwords where you don't want the input to be visible when it's being typed.

And then finally each parameter can have a type.

You have simple ones like string, number or list but you also have AWS specific ones which allow you to specify a VPC from a list or subnets from a list and some of these can be populated so from the console UI perspective they're interactive based on the region and the account that you're applying the template within.

Now you're going to be getting some practical experience of working with parameters in a future demo video.

For now I just want you to have a basic awareness.

Now visually parameter architecture looks like this.

Parameters start by being defined within a cloud formation template and let's use this as an example.

I've defined two parameters here so instance type which is a string and it has a default of t3.micro together with a set of three allowed values.

I've also provided a description which makes it easier to use from the console UI and then second we have instance AMI ID which is a normal string type parameter with no allowed values so this is simple free text.

Now this example is part of a wider template which includes an EC2 logical resource so if we load this into cloud formation via the console UI then this is what we might see a user interface presentation of those parameters.

At this stage we enter values or we accept the default values and we move through the process of creating the stack.

Conceptually this means that the template defines things based on the resources declared within it and the interactive values provided via the parameters so both of these are combined and are used to create the stack.

It means that the stack creates physical resources based both on the logical resources and the effect on them which the parameters have.

In this case based on the parameter values we would create an instance with one of three sizes and use a certain Amazon machine image.

Now most of this applies to both template and pseudo parameters.

The thing unique to template parameters is that the personal process provides the values into cloud formation either explicitly or by implicitly accepting the defaults.

Pseudo parameters can be treated in the same way but they're provided by AWS so let's have a look at that visually.

Now we start off with a familiar architecture a cloud formation template is used to create a cloud formation stack.

The template could be using the template parameters I've just been talking about.

You don't have to pick one type over the other.

Template and pseudo parameters can be used in a complementary way.

With pseudo parameters what happens is that AWS make available parameters which can be referenced and these exist even if you don't define them in the parameters section of the template.

So conceptually think of these as being injected by AWS into the template and stack.

Now an example of a pseudo parameter is AWS double colon region and the value of this parameter always matches whichever region a template is being applied in to create a stack.

In this example US - East - 1.

Other pseudo parameters include AWS double colon stack ID which matches the unique ID of the stack, AWS double colon stack name which matches the name on the stack and AWS double colon account ID which is populated with the account ID of the account that the stack is being created in.

So pseudo parameters think of them like template parameters but instead of being populated by a human or a process when creating the stack they're populated by AWS.

Now both types of parameters are useful in ensuring that a template is portable and can adjust based on input from the person or process creating the stack.

Static templates are much less flexible and this functionality goes a long way to removing the negative aspects of static templates.

From a best practice perspective you should aim to minimize the number of parameters which you have which require explicit input.

Now this means wherever possible using defaults and where possible getting values from AWS rather than whoever is implementing the stack.

In the videos which follow as well as learning more about the features of cloud formation which help with template portability you're going to get the chance to experiment with all of those features in some demos.

I'm introducing the theory first and then you'll get the chance to experience it yourself.

Now with that being said that's everything that I wanted to cover in this video so go ahead and complete the video and when you're ready I look forward to you joining me in the next.

Welcome back and in this video I want to cover two things which are at the core of CloudFormation as a product.

Physical resources and logical resources.

In covering both of those you're also going to be learning about templates and stacks.

So this will be a good video to cover the basics of CloudFormation.

Now we've got a lot to cover so let's jump in and get started.

CloudFormation begins with a template which is a document written in either YAML or JSON, both of which you should now have an awareness of.

And defined within a CloudFormation template are logical resources.

Think of logical resources as what you want to create but not how you want them created.

When using CloudFormation you focus on the what and let CloudFormation deal with the how.

CloudFormation templates can be used to create CloudFormation stacks.

And a template can be used to create one stack, a hundred stacks or twenty stacks in different regions.

The idea is that one template defines what resources you want.

And defining good templates means a template can be used many times in many accounts in many regions.

And we refer to that as a portable template.

The initial job of a stack is to create physical resources based on the logical resources defined within the template.

For every logical resource in a template when a stack is created a physical resource is also created.

If a stack's template is updated in some way and then the stack itself is updated the physical resources are also changed.

The stack keeps the logical and physical resources in sync.

If a stack is deleted then normally the physical resources are also deleted.

So think about CloudFormation as a product which looks at a template specifically the logical resources within a template.

And then it creates, modifies or deletes physical resources as required.

So visually it looks like this.

This is a CloudFormation template and this one has been written using YAML.

The template contains logical resources.

In this example instance is the name of the logical resource and this is the type.

So AWS double colon EC2 double colon instance.

Now logical resources are generally going to have properties which are used by CloudFormation when configuring the actual physical resources.

In this example this sets the Amazon machine image to use, the type of the instance and the SSH key pair to use when connecting to the instance.

So the collection of logical resources and other things which I'll be covering in future videos is called a CloudFormation template.

And this template can be used to create one or many CloudFormation stacks.

And a stack when created also creates physical resources based on the logical resources.

So this means because we've set the AMI to use in the template and the SSH key to use these will be used when creating the physical resource.

In this case an EC2 instance.

So this physical EC2 instance is a representation of the logical resource defined in the CloudFormation template.

Now the stack will also react to template changes to update or delete physical resources as required.

Once a logical resource defined inside the CloudFormation template moves into a create complete state, meaning that the physical resource has been created, then the logical resource can be referenced by other logical resources to retrieve various physical configuration elements or IDs.

For example in this case the physical machine ID of the EC2 instance.

So in summary logical resources are contained inside CloudFormation templates.

CloudFormation templates are used to create CloudFormation stacks and the stacks job is to create, update or delete physical resources based on what's contained in that template.

CloudFormation as a product aims to keep the two in sync, so physical and logical resources.

So when you use a template to create a stack, CloudFormation will scan the template and create a stack with logical resources inside and then create physical resources which match those logical resources.

If you update the template then you can use it to update that same stack.

When you do that the stack's logical resources will change, either new logical resources will be added or existing ones are updated or deleted and CloudFormation will perform the same actions on the physical resources.

So adding new ones, updating existing ones or removing physical resources entirely.

If you delete a stack its logical resources are also deleted which causes it to delete the matching physical resources.

CloudFormation is a really powerful tool which you'll be using extensively in the real world and this is the same whether you're a solutions architect, a developer or an engineer.

I use CloudFormation constantly in all of the AWS courses that I create and so by taking the courses you'll be gaining a lot of practical and theory understanding of how CloudFormation works.

Now if you're taking any of my courses with my CloudFormation mini deep dive then you'll be learning even more.

By talking about every important aspect of CloudFormation that's relevant for the course that you're taking as well as giving you plenty of practical examples.

CloudFormation lets you automate infrastructure.

Imagine that you host WordPress blogs.

You can use one template to create one, ten, a hundred or more deployments rather than having to create a hundred individual sites.

CloudFormation can also be used as part of change management.

You can store templates in source code repositories, add changes and get approval before applying them.

Or they can be used to just quickly spin up one-off deployments and if you're taking any of my AWS courses you'll be seeing that I'll be using CloudFormation extensively as part of any of the practical demo lessons in the course.

We'll be using templates to spin up any of the infrastructure that will support the demo lesson that you're going to be taking.

Now that's all of the theory that I wanted to cover about physical and logical resources within CloudFormation.

It is a fairly theoretical topic but you need to understand what a physical resource is, what a logical resource is and how the two relate together as far as they're used within CloudFormation.

Now at this point that's everything that I wanted you to cover in this video so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk briefly about Amazon Guard Duty.

Now this is something which you only need detailed knowledge of for the security specialty stream of training.

Now I'll try to keep this lesson as efficient as possible so let's jump in and get started.

Now it's important at the outset that you know what Guard Duty is and what makes it special.

So it's a security service but specifically it's a continuous security monitoring service.

This means once enabled it's running all the time trying to protect your account and resources from any security issues.

Now the way that it works is that it can be integrated with supported data sources and I'll talk about this more on the next screen.

It's constantly reviewing those data sources for anything occurring within the account and it also uses artificial intelligence and machine learning plus threat intelligent feeds.

Now the aim of the product is to identify any unexpected or unauthorized activity on the account.

Guard Duty is doing this in an intelligent way so you aren't having to identify things you usually do or define what normal activity is.

It attempts to learn this on its own and using threat intelligence feeds it tries to spot odd or worrying activity as it occurs on the account.

Now you can influence this so white listing IPs and influencing what it sees as okay behavior but the whole point of the product is that on the whole it learns patterns of what happens normally within any managed accounts.

Now if it finds something which logically is called a finding then it can be configured to notify somebody or initiate an event driven process of protection and/or remediation.

Now this might be a lambda function performing some kind of remediation or an event driven workflow via cloud watch events but Guard Duty can be part of an automatic event driven security response and that's really cool.

What's even more awesome is that it actually supports multiple accounts via a master and member account architecture.

When you enable Guard Duty you're essentially making the account that you enable it in the master Guard Duty account and then you can invite other AWS accounts and if they accept they become member Guard Duty accounts meaning the product supports a single location for managing multiple AWS accounts.

Now architecturally the product looks like this.

First we have Guard Duty and Guard Duty receives logs from supported data sources.

At the time of creating this lesson this includes DNS logs from Route 53 showing DNS requests, VPC flow logs showing traffic metadata for any traffic flowing through a VPC, cloud trail event logs showing any API calls within the account, cloud trail management events which cover any control plane level events and then finally cloud trail S3 data events which cover any interactions with objects within S3.

Now all of those are ingested together with various threat intelligent feeds and are used to generate findings which show any unusual or unexpected behavior.

These findings can be sent to cloud watch events now known as event bridge which can be used to handle event driven notification and automatic remediation.

So event bridge can use S&S for notifications to any team members or external security management systems or it can invoke Lambda functions which can interact with AWS APIs, products and services to help automatically remediate any security issues maybe to add an explicit deny rule to a network ACL if there's a potential intrusion.

Now that's pretty much all you need to know for the exam and to get started using the product in the real world.

Now thanks for watching go ahead and complete this lesson and then when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video I'm going to be talking about Amazon Inspector.

Now this is a service which is really simple to use and it only features in a relatively minor way on most of the AWS exams.

So this is a fundamental video.

If appropriate for the course that you're taking, I'll be going into much more detail in separate videos.

For this video you just need to have a basic awareness of what this product does and how to use it effectively.

Now nearly all of my lessons contain visuals because I find this helps students to learn better.

But in this case Inspector is just one of those services which is easy to understand but very detailed in terms of what it does.

And unfortunately this means it's going to be a text heavy lesson.

So let's jump in and get started.

Amazon Inspector is a product designed to check EC2 instances, the operating systems running on those instances as well as container workloads for any vulnerabilities or deviations against best practice.

The idea is to run an assessment of varying lengths, say 15 minutes, 1, 8 or 12 hours and even 1 day and identify any unusual traffic or configurations which put applications on the instances, the instances themselves or containers at risk.

Now at the end of this process the product will provide you with a review of findings ordered by severity.

In the exam if you see anything about a security report then think Inspector.

But remember it's checking instances, their operating systems, containers and any other networking components involved.

Now Inspector can work with two main types of assessments.

A network assessment can be conducted without using an Inspector agent but adding an agent provides additional richer information.

It can also run a network and/or host assessment which does use an agent.

The host assessment looks at OS level vulnerabilities and this needs access to inside of the instance, so the instance OS and this requires an agent.

With Inspector rules packages determine what is checked.

The first package, network reachability which can be done with no agent or with an agent for additional rich information.

This checks how an instance or group of instances is exposed to public networks, so it checks end-to-end reachability.

So EC2, application load balancers, Direct Connect, elastic load balancers, network interfaces, internet gateways, access control lists, route tables, security groups.

It even checks subnet and VPC configuration and even exposure from virtual private gateways and any VPC peering.

The network reachability rules package returns the following types of findings.

First, for recognized ports, so well-known ports, it confirms if the port is recognized with a listener, i.e. is it exposed to the public networks and is the operating system listening on that port, or recognized port no listener where it's exposed to the internet but with nothing listening, or if you don't use an agent, a recognized port which is exposed but there is no agent to check if the operating system is listening, and this is why using an agent always adds more information versus no agent.

Now lastly, it can identify any unrecognized ports which are exposed with listeners.

So for the exam, this is what the network reachability rules package does.

You might see that term, you need to know what it does, or it might request you to suggest a product which can do this type of analysis and then question whether an agent is required.

And so these are all key points to understand.

We also have rules packages which do require an agent, so host assessments, and all of these are really, really important to remember for the exam.

These are pure keywords, so easy to remember but massively important.

First, there is the common vulnerabilities and exposures or CVE package, and CVE is a database of known cyber security vulnerabilities, each of which is assigned a CVE number, and this package checks against those.

If you see CVE in the exam, think Inspector.

And a report will include any CVE IDs for anything found on the instances or containers.

Next, we have the Center for Internet Security or CIS Benchmarks.

The formal definition is the CIS Security Benchmark Program provides well-defined, unbiased, consensus-based industry best practices to help organizations assess and improve their security.

This rules package checks against that.

So again, if you see CIS as an exam question, think Inspector.

Then finally, we have Security Best Practices for Inspector, which is just a collection of best practices provided by Amazon, including things like disabling root login over SSH, using only modern version numbers for SSH, password complexity checks, and permissions on certain folders.

Again, if you see anything of this nature in the exam, think Inspector.

And that really is everything that you need to know at this fundamental level for this product.

Again, if you're studying for a particular exam which requires more information, I will have additional videos covering everything else in depth.

This is just a fundamental 101 level lesson.

Now, you'll know by now I do hate teaching based on just keywords, but this is one of those outliers where you don't really need to know all of the details.

But I don't want you dropping exam marks because you don't know any of these really valuable keywords.

And again, I'm just going to repeat this one more time.

If applicable for the course that you're studying, I'll be covering Inspector in much more detail in other dedicated lessons.

For now, though, that is everything I wanted to cover.

So go ahead and complete this video.

And when you're ready, I'll look forward to you joining me in the next.

Welcome to this lesson where we're going to be covering Amazon Macie.

Now we have a lot to cover so let's jump in and get started.

So what is Macie?

Well, it's a data security and data privacy service.

You'll understand now the architecture of the simple storage service known as S3.

It's one of AWS's most popular services and it can host huge quantities of large or small objects at scale.

It can also be made public and for some time it's been a constant source of risk within organizations because of the fact that data can be leaked if the service is misconfigured.

So Macie is a service which can be used to discover, monitor and protect data which is stored within S3 buckets.

It's critical if an organization wants to control the security of its data that it needs to have an awareness of where that data is and what exactly it contains.

So once enabled and pointed at buckets within your AWS account or AWS accounts, Macie can get to work discovering data and this might mean data which is classed as personally identifiable information or PII or personal health information known as PHI as well as financial data and many other types of data.

Now these high level categories include a huge range of data which you personally will have day to day familiarity with.

Things like AWS access keys, SSH keys, PGP keys or bank account numbers, credit card numbers or expiry dates, health insurance numbers, birth dates, drivers license numbers, national insurance numbers, passport numbers, addresses and much more.

It's the first job of Macie to identify and inventory this data.

So by using Macie you'll know what you have, what it contains and where it is.

Now the way that it does this is using data identifiers.

Think of these like rules which your objects and their contents are assessed against and there are two types of data identifiers.

Managed data identifiers and custom data identifiers.

Now managed data identifiers are built into the product.

They use a combination of criteria and techniques including machine learning and pattern matching to analyse the data that you specify.

They're designed to detect sensitive data types for many countries and regions including multiple types of personally identifiable information, personal health information and financial data.

And this type of identifier can be used to detect almost all common types of sensitive data that you might need to manage within your organisation.

Now you can also build custom data identifiers for your business.

These are proprietary so you can look for specific data which your business needs to identify and control.

An example you might use a regular expression known as a reg X to search for certain patterns of specific text within your business.

Maybe employee IDs or performance reports.

With Macy you create discovery jobs which use these identifiers and look for anything matching on buckets.

If anything is found these jobs generate findings and you can view these findings interactively or they can be used as part of integrations with other AWS services.

For example security hub or finding events can be generated and passed into EventBridge and then they can be used for automatic event driven remediation.

So it's a super powerful architecture.

Now one final thing which you need to understand before we review the architecture visually is that Macy uses a multi account architecture.

One account is the administrator account and that can be used to manage Macy within member accounts.

And this multi account structure can be done either using AWS organisations or by explicitly inviting accounts.

And once invited buckets across all accounts within the Macy organisation can be evaluated in the same way.

Now let's just take a second to review the architecture visually.

We start with one or more S3 buckets and then the Macy service itself and then we create a discover job.

And within the discover job we can specify which buckets we want to analyse which means detecting and classifying data within those buckets.

The discovery job has a schedule so this controls when it runs and how frequently it runs and then the job uses a combination of managed data identifiers and custom data identifiers.

And these are the things which actually identify and classify the types of data that Macy is locating.

So it's these things which are the important part of the whole process.

Now as an output to the discovery job findings are generated and these can be viewed either using the console interactively or and this is the more common use case.

They can be used with a vent bridge in the form of event findings generation which can then be delivered to other AWS services.

And this is commonly used for integration or for event driven remediation in this example where a Lambda function can receive the event and can perform some kind of automatic fix based on the finding.

So at a high level that's how the architecture looks.

And before we finish up with this lesson I want to explore a number of other important elements of the service and we're going to start with looking in more detail at the managed and custom data identifiers.

To discover sensitive data within Amazon Macy you create and run data discovery jobs.

A data discovery job analyzes objects within S3 buckets to determine whether the objects contain sensitive data.

And the way that it does this is via data identifiers.

First we have managed data identifiers and these are created and managed by AWS.

And as I mentioned earlier in this lesson they can be used to identify a growing list of sensitive data types.

Now I've included a link attached to this lesson which details the full range of data which is matched by this type of identifier.

But it's things like various credentials, financial data, credit cards, bank details and more.

Things like health data or anything personally identifying such as addresses, passports, drivers licenses and much much more.

It's a pretty comprehensive list so it's worth checking out the link that's included with this lesson which gives a full overview.

In addition to this anyone can create custom data identifiers.

Now the foundation of these are regular expressions which define a pattern to match within data.

This one for instance matches any data which contains the letters A through to Z and then a dash and then eight digits.

Anything that you can define using regular expressions you can match using custom data identifiers.

And these are generally used for data patterns which are custom to your organization as with this example of an employee ID.

You can optionally add keywords to custom data identifiers which must occur within a definable proximity to the pattern matched by the regular expressions.

And this definable distance is called the maximum match distance.

And then finally you can also include ignore words.

So if the regex match is something but an ignore word is there in addition it's ignored and doesn't match.

So keywords, maximum match distance and ignore words are all refiners.

They help you start with a regex pattern but influence how something is classified based on those refinements.

So these identifiers run in addition to built in checks that Macy performs and then findings are generated.

And Macy will produce two types of findings.

Policy findings and sensitive data findings.

Macy generates policy findings when the policies or settings for an S3 bucket are changed in a way that reduces the security of the bucket or its objects but crucially after Macy is enabled.

For example if the default encryption on a bucket was enabled when you enabled Macy and then default encryption is later disabled on that bucket then this is highlighted as a policy finding.

So that's an example of a policy finding.

Macy generates the other type of finding which is a sensitive data finding when it discovers sensitive data in S3 objects that you configure it to analyze.

And it determines what is sensitive data based on the jobs and identifiers which you configure and which I've just stepped through.

So some examples of policy findings are S3 block public access disabled which is triggered if the block public access settings on a bucket are disabled.

Another is S3 bucket encryption disabled which is triggered logically when encryption on a bucket is disabled.

Another is S3 bucket public which is triggered when a bucket policy or ACL changes are made which make a bucket public and another is S3 bucket shared externally and this is triggered when a bucket policy or ACL allows an AWS account other than those within the Macy organization access to this bucket.

So these are all policy changes which Macy decides reduce the security of a bucket or objects in that bucket and so trigger policy findings.

So these are called policy findings and there are more of these and I've included a link attached to this lesson which details all of them and that's really worth a look through just to become familiar with all of the different things that Macy can identify.

Now examples of sensitive data findings include these and it's worth pointing out that there are many more of them.

Again I've included a comprehensive list which is attached to this lesson but for now let's just focus on these important examples.

First we have S3 object credentials and this matches any exposed SSH keys or AWS access keys that Macy can locate.

We've also got S3 object custom identifier and this matches anything defined within custom data identifiers.

We have S3 object financial which matches credit card numbers or bank account numbers and much more.

We have S3 object multiple which occurs when more than one thing is identified.

We have S3 object personal which covers personally identifiable information such as full names, mailing addresses, personal health information such as health insurance or medical identification numbers or combinations of those.

Now this isn't an exhaustive list.

Again I've included a link attached to this lesson which gives you a full overview.

And that at a high level is Macy.

It's a useful tool which you'll need to understand for the exam.

If you see any questions regarding the classification of data within S3 so identifying data, discovering data or reacting to sensitive data automatically then Macy is probably the product to use.

Now that's everything I wanted to cover within this theory lesson.

If you're doing any of my courses where practical knowledge of Macy is required then there's going to be a demo lesson immediately following this one.

If not then this theory is all that you'll need.

So at this point this is the end of the lesson.

Thanks for watching.

Go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about AWS config.

Now let's just jump in and get started because we've got a lot to cover.

AWS config is an interesting service because people often misunderstand what it does.

This is especially important within exam situations where you don't have the benefit of Google and have to make architectural decisions quickly.

Now AWS config has two main jobs.

Its primary function is to record changes over time on resources within an AWS account.

Once enabled, the configuration of every resource in the account is monitored.

Every time a resource's configuration changes, a configuration item is created which stores the configuration of that resource at a specific point in time.

The information which is stored is the configuration of the resource, the relationship to other resources and who makes any changes.

So for example, if you had a security group attached to an instance and you added a rule to that security group, then it would track the pre-change state, the post-change state, the fact that you changed it and the fact that it was attached to that EC2 instance.

Now this makes AWS config great for auditing changes and for checking if resources are compliant with standards defined by your organization.

The most important thing to understand about AWS config is that it doesn't prevent changes happening.

It's not a permissions product or a protection product.

Even if you define standards for resources, it can check compliance against those standards but it doesn't prevent you from breaching those standards and creating non-compliant resources.

An example of compliance might be a certain set of allowed ports within security groups.

You can add additional ports exposing an instance to a certain amount of risk.

Now AWS config won't stop you but that non-compliance, that additional port will be identified.

Now config is a regional service so when enabled it monitors changes within a particular AWS region in a particular AWS account but it can be configured for cross-region and cross-account aggregation.

It can also generate notifications via SNS and it can generate events via EventBridge and Lambda when resources change in terms of their compliance state so while AWS config won't prevent you changing something it can be used for automatic remediation.

Now the product stores all of the configuration data and changes in a consistent format within the S3 config bucket and the product allows you to access that data so all of the configuration history of all of the resources and you can interact with them directly from that bucket or using the AWS config APIs.

Now there are two sides to AWS config, the features which are standard and the parts of the product which are optional.

Now the standard part is on the left and the optional part is on the right.

So starting on the left we have some account resources and we have AWS config.

To use the product we have to enable it and this enables the recorder functionality and this takes config information of all the resources and stores them in an S3 bucket, the config bucket and this is all part of the standard functionality provided by the product.

Now you could just enable all of this functionality and leave this as it is.

This would allow you to record and review all changes to resources over time.

Every time a change happens a configuration item would be generated and all of these for all resources would be stored in a standard format in the config bucket.

But we can do a lot more with the product and this is where the real power of AWS config comes from because we can use config rules.

Now config rules are either AWS managed ones or you can define your own which uses Lambda.

What happens is that these rules evaluate resources against a defined standard.

Resources based on these rules are either compliant or non-compliant based on if they meet criteria specified within the config rule.

Now custom rules use Lambda to evaluate if resources match criteria.

The Lambda function does the evaluation using whatever things that you can code and then returns information back to AWS config.

AWS config can then notify or work with other products for automatic remediation.

For example, it can use SNS to send either a stream of changes or compliance notifications and these will either go to human operators or other applications to deal with.

In addition though you can integrate AWS config with EventBridge.

So for any changes in the state of config rules whenever anything becomes compliant or non-compliant this event can be sent to EventBridge and then EventBridge can be used to invoke Lambda functions to perform automatic remediation of any changes.

So to fix the problems automatically.

Now this isn't strictly part of AWS config.

You're essentially using EventBridge to send any events from AWS config to targets to perform this automatic remediation.

You can also fix these type of config changes using SSM.

So AWS config can integrate with systems manager and apply fixes to remediate any issues.

But Lambda can be more flexible for account level things whereas SSM can be effective for anything relating to the configuration of instances.

Now that's all of the theory that I wanted to cover in this lesson.

Go ahead and complete this lesson and then when you're ready I look forward to you joining me in the next lesson.

Welcome back and in this lesson I want to talk about CloudHSM.

Now this is a product which is similar to KMS in terms of the functionality which it provides, in that it's an appliance which creates, manages and secures cryptographic material or keys.

Now there are a few key differences and you need to know these differences because it will help you decide on when to use KMS and when to use CloudHSM.

And you might face an exam question where you need to select between these two.

So let's jump in and get started.

Now I promised you at the start of the course I wouldn't use facts and figures in lessons unless absolutely required.

You shouldn't have to remember lots of different facts and figures unless they influence the architecture.

Now this unfortunately is going to be one of the lessons where I do have to introduce some keywords that you simply need to remember.

Because in this lesson the detail, the difference between CloudHSM and KMS really matters.

Now let's start by quickly talking about KMS.

KMS is the key management service within AWS.

So it's used essentially for encryption within AWS and it integrates with other AWS products.

So it can generate keys, it can manage keys, other AWS services integrate with it for their encryption.

But it has one security concern, at least if you operate in a really demanding security environment.

And that's that it's a shared service.

While your part of KMS is isolated, under the covers you're using a service which other accounts within AWS also use.

What's more, while the permissions within AWS are strict, AWS do have a certain level of access to the KMS product.

They manage the hardware and the software of the systems which provide the KMS product to you as a customer.

Now behind the scenes KMS uses what's called a HSM which stands for Hardware Security Module.

And these are actually industry standard pieces of hardware which are designed to manage keys and perform cryptographic operations.

Now you can actually run your own HSM on-premise.

Cloud HSM is essentially a true single tenant HSM that's hosted within the AWS cloud.

So if you hear the term HSM mentioned, it could refer to both Cloud HSM which is hosted by AWS or an on-premise HSM device.

Now specifically focusing on Cloud HSM, AWS provision it and they're responsible for hardware maintenance.

But they have no access to the part of the unit where the keys are stored and managed.

It's actually a physically tamper resistant piece of hardware.

So it's not something that they can gain access to.

Generally if you as the customer lose access to a HSM, that's it, game over.

You can reprovision them but there's no easy way to recover data.

Now there's actually a well-known standard for these cryptographic modules.

It's called the Federal Information Processing Standard Publication 140-2.

You can easily determine the capability of any HSM modules based on their compliance with this standard.

And I've included a link in the lesson description with additional information.

But Cloud HSM is FIPS 140-2 Level 3 compliant and it's the Level 3 which really matters in the context of this lesson.

KMS in comparison is overall 140-2 Level 2 compliant and some of the areas of the KMS product are also compliant with Level 3.

Now this matters.

This is really important.

If you see an exam question or if you're in a real world production situation which requires 140-2 Level 3 overall, then you have to use Cloud HSM or your own on-premises HSM device.

And that's a fact that you really need to remember for the exam.

Another important distinction between KMS and Cloud HSM is how you access the product.

With KMS, all operations are performed with AWS standard APIs and all permissions are also controlled with IAM permissions.

Now Cloud HSM isn't so integrated with AWS and this is by design.

With Cloud HSM, you access it with industry standard APIs.

Now examples of this are PKCS 11, the JCE extensions or the CryptoNG extensions.

And I've highlighted the keywords that you should try to build up an association with Cloud HSM.

So if you see any of these keywords listed in the exam or in production situations, then you know you need a HSM appliance, either on-premise or Cloud HSM hosted by AWS.

Now it used to be that there was no real overlap between Cloud HSM and KMS.

They were completely different.

But more recently, you can use a feature of KMS called a custom key store.

And this custom key store can actually use Cloud HSM to provide this functionality, which means that you get many of the benefits with Cloud HSM together with the integration with AWS.

So when you're facing any exam questions, you still should be able to look for these keywords to distinguish between situations when you use KMS versus Cloud HSM.

Now just to summarize before we move on from this screen, I want you to focus on doing your best to remember all of the three key points that are highlighted with the exam power-up icon.

If you can remember those, then you should be in a really good position to determine whether to use KMS or Cloud HSM within exam questions.

Now I want to look at the architecture of Cloud HSM as a product, and I think it's best that we do that visually.

Now architecturally, Cloud HSMs are not actually deployed inside a VPC that you control.

They're deployed into an AWS managed Cloud HSM VPC that you have no visibility of.

So architecturally, this is how that looks.

So on the left, we've got a customer managed VPC.

On the right, we've got the Cloud HSM VPC that's managed by AWS.

We're using two availability zones, and inside the customer managed VPC, we've gone ahead and created two private subnets, one in availability zone A and one in availability zone B.

Now inside the Cloud HSM VPC, to achieve high availability, you need to deploy multiple HSMs and configure them as a cluster.

So a HSM by default is not a highly available device.

It's a physical network device that runs within one availability zone.

So in order to provide a fully highly available system, we need to create a cluster and have at least two HSMs in that cluster, one of them in every availability zone that you use within a VPC.

Now once HSM devices are configured to be in a cluster, then they replicate any keys, any policies, or any other important configuration between all of the HSM devices in that cluster.

So that's managed by default, by the appliances themselves.

That's not something that you need to configure.

So the HSMs operate from this AWS managed VPC, but they're injected into your customer managed VPC via elastic network interfaces.

So you get one elastic network interface for every HSM that's inside the cluster injected into your VPC.

Once these interfaces have been injected into your customer managed VPC, then any services which are also inside that VPC can utilize the HSM cluster by using these interfaces.

And if you want to achieve true high availability, then logically instances will need to be configured to low balance across all of the different interfaces.

Now also in order to utilize the cloud HSM devices, then a client needs to be installed on the EC2 instances, which are going to be configured to access the cloud HSM.

So this is a background process known as the cloud HSM client.

And this needs to be installed on the EC2 instance in order for it to access the HSM appliances.

And then once the cloud HSM client is installed, then you can utilize industry standard API's such as PK, CS11, JCE and crypto NG to access the HSM cluster.

Now a really important thing to understand about cloud HSM, because this is a distinguishing factor between it and KMS, is that while AWS do provision the HSM, they're actually partitioned and they're tamper resistant.

So AWS have no access to the area of the HSM appliances which store the keys.

Only you can control these.

You manage them, you're responsible for them.

Now AWS can perform things like software updates and other maintenance tasks, but these don't take place on the area of the HSM which is used to perform cryptographic operations.

Only you as an administrator or anyone that you delegate that to has the ability to interact with the secure area of the HSM devices.

Now before we finish this lesson, there are a few more things that I want to cover.

So these are points that I think you should be aware of.

So some of these are use cases, some of these are limitations that will help you select between using cloud HSM and using something like KMS.

So first, by default there's no native integration between cloud HSM and any AWS products.

So one example of this is that you can't use cloud HSM in conjunction with S3 server-side encryption.

That's not a capability that it has.

Cloud HSM is not accessed using AWS standard APIs at least by default and so you can't integrate it directly with any AWS services.

Now you could, for example, use cloud HSM to perform client-side encryption.

So if you've got an encryption library on a particular local machine and you want to encrypt objects before you upload them to S3, then you can use it to perform that encryption on the object before you upload it to the S3 service.

But this is not integrated with S3.

You're just using it to perform encryption on the objects before you provide them to S3.

Now a cloud HSM can also be used to offload SSL or TLS processing from web servers.

And if you do that, then the web servers can benefit from A, not having to perform those cryptographic operations, but also the cloud HSM is a custom designed piece of hardware that accelerates those processes.

So it's much more economical and efficient to have a cloud HSM device performing those cryptographic operations versus doing it on a general purpose EC2 instance.

So that's something that a cloud HSM can do for you, but KMS natively cannot.

Now other products that you might use inside AWS can also benefit from cloud HSM, products which are able to interact using these industry standard APIs.

And this includes products like Oracle databases.

So they can utilize cloud HSM for performing transparent data encryption or TDE.

So this is a method that Oracle has for encrypting data that it manages on your behalf.

And it can utilize a cloud HSM device to perform the encryption operations and to manage the keys.

Now this does mean that because a cloud HSM device is something that's entirely managed by you, you're the only entity that initially starts off with access to be able to interact with the encryption materials.

So the keys, it means that if you use a cloud HSM and integrate it with an Oracle database, then you're doing so in a way which means that AWS have no ability to decrypt that data.

And so if you're operating in a highly restricted regulatory environment where you really need to use strong encryption and verify exactly who has the ability to perform encryption operations, then generally cloud HSM is an ideal product to support that.

And then lastly in a similar way, cloud HSM can also be used to protect the private keys for a certificate authority.

So if you're running your own certificate authority, you can utilize cloud HSM to manage the private keys for that certificate authority.

Now just to summarize at this point, the overall theme is that for anything which isn't specific to AWS, for anything which expects to have access to a hardware security module using industry standard APIs, then the ideal product for that is cloud HSM.

For anything that uses standards for anything that has to integrate with products which aren't AWS, then cloud HSM is ideal.

For anything which does require AWS integration, then natively cloud HSM isn't suitable.

If FIPS 140-2 Level 3 is mentioned, then it's cloud HSM.

If integration with AWS is mentioned, then it's probably going to be KMS.

If you need to utilize industry standard encryption APIs, then it's likely to be cloud HSM.

Now that's everything that we need to cover.

I just wanted you to be able to handle any curveball HSM style questions that you might encounter in the exam.

So thanks for watching, go ahead and complete this video and then when you're ready, I'll look forward to you joining me in the next one.

Welcome back and in this lesson I want to talk about AWS Shield, which is an essential tool to protect any internet connected environment from distributed denial of service attacks.

Now it's important for the exam, but especially so for the real world.

So let's jump in and get started.

So AWS Shield actually comes in two forms, Shield Standard and Shield Advanced.

Both of them provide protection against DDoS attacks, but there's a huge difference in their respective capabilities.

First, Shield Standard is free for AWS customers, whereas Shield Advanced is a commercial extra product, which comes with additional costs and benefits, which I'll detail later in this lesson.

The product protects against three different types or layers of DDoS attack.

Now I've covered these in the DDoS lesson in the technical fundamental section of the course, but as a reminder, these categories are Network Volumetric Attacks, so these are things which operate at layer three of the OSI 7 layer model, and these are designed to simply overwhelm the system being attacked, so to direct as much raw network data at a target as possible.

Next, we have Network Protocol Attacks, such as SYNFLUDS, and these operate at layer four of the OSI model.

Now there are various types of protocol attack, but one common one is to generate a huge number of connections from a spoofed IP address and then just leave these connections open, so never terminating them, and while the CPU memory and data resources of the target will be fine, its ability to service real connections will be impacted by the huge volume of fake ones.

To understand this, imagine a call centre where people call up and just leave the phone line silent.

The operators won't be doing anything, but there won't be capacity for new calls to be answered.

Now Network Protocol Attacks can also be combined with volumetric attacks, but by default, you should view these as two different things.

Lastly, we have Application Layer Attacks, which operate at layer seven, for example, Web Request Floods.

Imagine you have a part of your web app which allows searchers.

Think of something like this which lets you search for every cat image in the world ever.

From the perspective of the attacker, this uses almost no resources to run.

It can be done hundreds, thousands or millions of times per second.

But from the perspective of the system being attacked, this might take two to three seconds to return data, maybe even more.

And so it's possible to de-doss a system by using the application as intended, where certain parts of the application are cheap to request, but expensive to deliver the result.

So those are the types of things which SHIELD protects against.

Now I want to spend a little time delving deeper into the capabilities of SHIELD Standard and Advanced, together with the differences.

And I want to focus on when you might pick one versus the other.

So let's start with SHIELD Standard.

SHIELD Standard, as I mentioned earlier, is free for all AWS customers, so you benefit from its protection automatically without you having to do anything.

The protection is at the perimeter of the network, which can either be in your region, meaning as data flows into a VPC, or it can be at the edge of the AWS network if you use CloudFront or Global Accelerator.

SHIELD Standard protects against common network or transport layer attacks.

So that's attacks at layer three or four of the OSI seven layer model.

Now you get the best protection if you use Route 53, CloudFront or Global Accelerator.

Now SHIELD Standard doesn't provide much in the way of proactive capability or any form of explicit configurable protection.

It's just there working away in the background.

Now that's the foundation, the baseline of the product.

Now let's look at what extra things SHIELD Advanced offers.

So SHIELD Advanced, as a starting point, is a commercial product.

In fact, it costs $3,000 per month per organization.

Now this is important, it's not per AWS account.

If you have multiple accounts where you're wanting the advanced level of protection that SHIELD Advanced offers, then just make sure they're in the same AWS organization and you can share the one single investment.

Now the cost while it is per month is part of a one year commitment.

So at 3K per month, this means $36,000 per calendar year.

And there's also a charge for data out for using the product.

Now SHIELD Advanced protects more than standard.

It covers CloudFront, Route 53, Global Accelerator, anything associated with elastic IPs, for example EC2.

It also covers application, classic and network load balances.

It's a comprehensive set of DDoS protections for your network perimeter.

Now what's really important to understand as a concept is that the protections offered by SHIELD Advanced are not automatic.

You need to explicitly enable protections, either in SHIELD Advanced or as part of AWS Firewall Manager when using SHIELD Advanced policies.

It's an explicit act, remember that.

You might find a question on the exam where you need to answer whether these protections require explicit configuration or they happen in the background.

Now SHIELD Advanced offers two other really important benefits and it's important to understand that these are not technical functionality differences, but they're important nonetheless.

First you get cost protection.

And this means that if you as a customer incur any costs for any attacks which should be mitigated by AWS SHIELD Advanced, but aren't, then you're protected against those costs.

And an example of this might be EC2 scaling events caused by excessive load.

Now there are restrictions, it needs to be something SHIELD Advanced should cover and you should have enabled the coverage on that resource.

Now I've included a link attached to this video which covers this particular feature in much more detail.

You don't need to understand the detail for the exam, but for the real world it's good knowledge to have.

Now the other benefit is a proactive style of management as well as access to the AWS SHIELD Response Team known as SRT.

With proactive management, the SHIELD Response Team contacts you directly when the availability or performance of your application is affected because of a possible attack.

And this provides the quickest level of response.

It allows the SHIELD Response Team to begin troubleshooting even before they've established contact with you, the customer.

Now to use this you need to provide your contact details in advance and enable the feature.

And when you do, the SHIELD Response Team will contact you when any attacks are detected.

You can also contact the SHIELD Response Team to log support tickets.

And the SLA for this depends on your support plan.

It might be one hour or 15 minutes.

These are all things that you need to think about and decide upon up front.

Now let's step through some of the technical ways in which SHIELD Advanced helps us.

The first unique feature of SHIELD Advanced is the integration with the web application firewall.

SHIELD Advanced uses the web application firewall to implement its protection against layer 7 attacks.

And if you have a SHIELD Advanced subscription, this includes basic WAF fees to implement these protections.

This is one of the differences in feature benefits which SHIELD Advanced provides over SHIELD Standard.

And so it's an important one to keep in mind.

Another benefit that SHIELD Advanced provides is advanced real-time metrics and reports for DDoS events and attacks.

And these can be accessed via the SHIELD Advanced Console or APIs and via CloudWatch.

Now, if you have a business need for SHIELD Advanced, if you can justify the cost, you're also going to have a need for this enhanced level of visibility.

So this is another one to keep in mind.

You also have health-based detection and this is using Route 53 health checks to implement application-specific health checks.

Now, this allows you to reduce any false positives detected by AWS SHIELD.

And it's used alone or in combination with the proactive engagement team to provide faster detection and mitigation of any issues.

Health-based detection is actually a requirement for using the proactive engagement team.

Again, another important thing to remember.

Now, lastly, you also have protection groups and you can use protection groups to create groupings of resources which SHIELD Advanced protects.

You can define the criteria for membership in a protection group so any new resources are automatically included.

And with these groups, you gain the ability to manage protection at a group level versus a resource level which can significantly decrease the admin overhead of using the product.

Now, at this point, that's everything I wanted to cover about AWS SHIELD at a high level.

If the topic that you're studying requires any additional detail, there will be additional deep dive lessons.

If not, don't worry, this is everything that you need to know.

But at this point, that's the end of this video.

So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this video I want to talk about the web application firewall known as WAF.

Now this is a key part of the AWS network and network security product set so let's jump in and get started.

Now WAF is AWS's implementation of a layer 7 or application layer firewall which we talked about in a previous video.

That means a firewall which is capable of understanding layer 7 protocols such as HTTP and HTTPS.

Now before I talk in detail about the features of the product I want to visually step through how a WAF architecture might look.

Now the example I'll be using is relatively complex.

They can range from fairly simple through to this type of example which is relatively complex involving event-driven security response.

So we start with an AWS environment and we decide to use a web application firewall which protects web resources which supports WAF and these include CloudFront, application load balancers, AppSync and API Gateway.

Now WAF is the product but the actual unit of configuration within the product is known as the web access control list known as web ACL and it's this which is used by WAF and also associated with the various supported services.

So you would associate a web ACL with a CloudFront distribution and this would result in the CloudFront distribution being protected by WAF.

So WAF can protect global services such as CloudFront but also regional resources such as application load balancers, API gateways and AppSync and you need to configure this when you create the web ACL essentially creating it in a region rather than globally as is the case for CloudFront.

Now within a web ACL you have rule groups and rules and I'll be talking more about how this is architected later in this video.

At a high level this might be things like AWS managed rules or simple allow or deny lists.

It might cover things like SQL injections or cross-site scripting attacks, HTTP floods or might relate to things like IP reputation and even protect against known botnets.

It's these rule groups and rules which control how the WAF product reacts to incoming traffic allowing connections from valid users while hopefully blocking those from bots and other attackers against your web resources.

Now this alone would be a super useful product which offers significant security benefits but when we combine this with other AWS products and architectures it can be even more useful.

You can obviously update the web ACLs manually based on human identified security events or risks but you could also do simple automated things such as using event bridge and scheduled rules to pass various publicly maintained IP lists to block known bad actors.

Now WAF does output logs and logging can be directed at S3 directly at cloud watch logs or kinesis firehose.

Now importantly if you want to react to logs quickly you shouldn't directly use S3 as these are delivered directly approximately every five minutes.

Firehose can be configured to put the logging data into any of its supported destinations including S3 and then all of these destinations can be integrated with an event driven security response architecture so using a combination of products such as S3 events, Lambda, Athena and EventBridge you can extract and identify intelligence to act on and then use this intelligence to update web ACLs to improve the security of the platform in a way which doesn't require humans.

So this type of architecture is based on taking basic WAF and creating a feedback loop to take data, identify actionable intelligence and then automate changes based on that intelligence.

And now that you have a visual idea about how a WAF implementation might look let's look at the raw features and how everything fits together.

I mentioned earlier in this video that the web access control list or web ACL is the main unit of configuration within WAF.

A web ACL is what controls if traffic is allowed or blocked.

The starting point of a web ACL is a default action which will either allow or block any traffic which isn't matched by the ACL.

Which one of these you pick depends on if you're using WAF to explicitly protect against certain exploits or if you want to only allow known good traffic through to your web resources.

Additionally a web ACL is created for either cloud front which is a global service or a regional service such as an application load balancer, API gateway or AppSync.

If you create a web ACL designed for a regional service then you have to define a region for the web ACL which matches the region that your services are located in.

Now web ACLs on their own don't do anything you have to add rule groups or rules and these are processed in order.

This order can be changed so you can move rules and rule groups around.

Now I'm going to talk about rule groups and rules in a second but conceptually rules have a certain compute requirement based on their complexity.

Web access control lists have a limit of how much compute requirements rules contained within them can use.

An AWS have a concept called web ACL capacity units or WCU and this is not to be confused with DynamoDB WCU which is right capacity units.

Web ACL capacity units are an indication at the complexity of rules and a web ACL has a default maximum of 1500 WCU that can be increased with a support ticket.

Web ACLs are the things which are associated with resources so you associate a web ACL for example with a cloudfront distribution and this association can take some time it depends on the service.

Cloudfront for example needs to update the distribution and then push this out to edge locations so this can take a fair bit of time.

Adjusting a web ACL which is already associated to resources is quicker so keep that in mind.

Now this is web ACLs at a high level these are the things which contain rules or rule groups and the things which are associated with resources.

Importantly the relationship is currently that a resource can have one web ACL but one web ACL can be associated with many resources.

Now also because of the global nature of cloudfront you can't associate a cloudfront web ACL with a regional resource or vice versa and web ACLs can currently not be used with AWS outposts.

Okay let's move on and talk about rule groups.

Rule groups as the name suggests are groups of rules they contain rules.

They're used by web ACLs they're a feature which allows grouped admin of rules.

They don't themselves have any default actions they're added to web ACLs and the web ACLs have the default action for anything not matched by rules either within groups or added directly to that web ACL.

Now rule groups are either managed by AWS or a marketplace vendor, yours so managed by you or service owned for example SHIELD or firewall manager owned groups.

AWS managed rule groups are mostly available for free for AWS WAF customers.

The AWS WAF bot control and fraud control account takeover protection rule groups do have additional fees and I'll be talking about this later in the video where I talk about pricing.

Now any rule groups obtained via the marketplace also generally have a subscription attached to them so you need to keep this in mind.

Rule groups can be reused within many web ACLs.

They're a separate entity and a web ACL can reference one or more rule groups.

When you create a rule group you define upfront the WCU capacity and the default maximum is the same 1500 WCU.

This indicates the amount of resources the rule group uses with its rules and it helps inform anyone using the rule group when they're building a web ACL.

So now that we've talked about rule groups which are essentially an admin or contain a concept and rule groups can be referenced from web ACLs let's talk in more detail about the rules themselves.

Now within WAF rules have a simple enough structure.

We've got type, statement and action.

The type of rule determines at a high level how it works.

The statement consists of one or more things which match traffic or not and the action is what WAF does if a match occurs.

Now rules are one of two types.

We've got regular and rate based.

Regular rules are designed to match if something occurs.

Rate based are designed to match if something occurs at a certain rate.

An example of this might be that you might have a rule to allow SSH connections from a certain IP address and this is an example of a regular rule but you might want another rule which allows you to do something if anyone attempted to connect via SSH save 5,000 times in a five-minute period because this would suggest a brute force attack.

So you need to understand the differences between regular and rate based rules.

Then you have the statement of a rule and this is the main part of a rule.

It defines what the rule checks for.

For regular rules think of this as a what.

What does the rule match against?

Examples might be incoming TCP port 80 or incoming SSH or it might be requests which have a certain HTTP header.

For rate based rules it's slightly different.

You're either going to be applying a rate limit on the number of connections for a source IP address or you're going to be applying a rate limit to connections which come from an IP address which also match certain criteria.

So for example you might want a rule to apply if a client makes 5,000 connections over a five-minute period or you might only want this to apply if 5,000 connections to SSH are made within a five-minute period.

Now in terms of criteria you can match against things like origin country, IP address, label and I'll talk more about this in a second.

Headers, cookies, query parameters, your IPath, query string, the body of a request or the HTTP method.

Now for the body it's important to understand that WAF is only checking the first 8,192 bytes.

Again remember this one for the exam.

Now depending on the criteria that you select from this list you can then choose how to match.

Examples include exact matches, starts with, ends with, contains and much more.

You can even match using regular expressions.

Now you can also have more than one statement.

Rules can have a single statement but also multiple and if you do have multiple you can choose whether to use and/or not conditions so you can define a pretty complex set of evaluation conditions.

Now next we have the rule action and for regular rules these can allow block count or run a capture.

For rate-based rules you can block count or run a capture.

Allow and block obviously affect whether traffic is allowed.

Count just counts the number of requests and records that data and capture runs a capture on the request so if a valid response is received it treats it as a count records it and continues processing and if the capture fails it's blocked and processing stops.

Now it makes sense that allow is not valid for rate-based rules since conceptually you want to do something if a rate is above a certain value and it doesn't make sense that that something is allow.

So remember with rate-based rules you're essentially wanting to perform an action if a rate is above a certain level.

So remember with rate-based rules you only have block count and capture you don't have allow.

Now you can also add custom responses as an optional extra.

If your action is block then this can be a custom response or a custom header.

For allows counts and capture this can be a custom header only.

This custom header means your application itself can react to traffic which has been matched and custom headers are prefixed with x-amzn-waf- and the header is used so that your application can react in some way to traffic which has been affected by a rule.

Now optionally labels can also be added.

Labels are internal to WAF but what it allows is multi-stage flows where one rule can add a label and whether another rule runs can be based on the label being present or not.

Labels as I just mentioned are internal to WAF only and they can be referenced from other rules within a single web ACL.

They don't persist outside of that.

Now importantly using labels relies on WAF not stopping processing and this is an important thing to understand with allow and block actions if a rule matches no further action occurs.

Processing for that bit of traffic on that web ACL is stopped.

For count and capture actions processing continues and this is where you typically use labels in follow-up rules which react in different ways based on that label being present.

Now let's finish up by talking about pricing.

With WAF you're charged a monthly price for every web ACL.

Now currently this is $5 per month and I've put an asterisk next to this because this is subject to change so don't be surprised if this specific value is different when you're watching this lesson.

Now also remember that web ACLs can be reused across different supported products and this is a monthly price per web ACL.

There's also a charge per rule on a web ACL and this is a monthly charge currently $1 per month but again this is subject to change and also you're going to be charged another monthly fee for every rule group or managed rule group that you add to your web ACL.

You've also got a charge for every request processed by a web ACL.

Now again currently this is $0.6 per month for every 1 million requests.

Now this charge is per web ACL so although you're only charged the single fee per web ACL and this can be reused across different products logically the more products that you use a web ACL on the higher the number of requests so this particular part of the pricing architecture will increase the more usage a web ACL has.

Now if you need to understand this in detail I do suggest using the AWS pricing calculator and I've linked this attached to this video.

This makes it really easy to just enter some values and see the true breakdown of using the WAF product.

Now in addition to these costs there are also optional security features which can be enabled on your web ACL and these are in the area of intelligent threat mitigation and these come with additional fees.

So first we have bot control and this comes with a monthly fee as well as a request based fee so this is a charge for every 1 million requests and again these are both subject to change they're accurate at the moment but don't be surprised if these prices are different when you're watching this video.

Next we have captures and there's a price per 1000 challenge attempts and again this is subject to change.

Next the fraud control and account takeover has a monthly charge and then a charge for every 1000 login attempts analyzed and lastly of course any marketplace rule groups that you choose to utilize will come with extra costs and these are all things that you need to keep in mind.

So that's the architecture and feature overview of the WAF product.

Now elsewhere in the course if appropriate there's going to be a demo where you can get experience of working with WAF in a practical sense.

If you don't need practical knowledge for the particular thing that you're studying then this is all the information that you require.

At this point though that's all of the theory that I want to discuss so go ahead and complete the video and when you're ready I look forward to you joining me in the next.

Welcome back and in this video I want to talk in general about application layer firewalls also known as layer 7 firewalls named after the layer of the OSI model that they operate at.

Now I want to keep this video pretty generic and talk about how AWS implement this within their product set in a separate video.

So let's just jump in and get started.

Now before I talk about the high level architecture and features of layer 7 firewalls, let's quickly refresh our knowledge of layer 3, 4 and 5.

So we start with a layer 3 and 4 firewall which is helping to secure the Categorum application.

Now this is accessed by millions of people globally because it's that amazing.

Now because this is layer 3 and 4, the firewall sees packets and segments, IP addresses and ports.

It sees two flows of communications, requests from the laptop to the server and then responses from the server back to the laptop.

Because this firewall is limited to layer 3 and 4 only, these are viewed as separate and unrelated.

You need to think of these as different streams of data, request and response, even though they're part of the same communication from a human perspective.

Now if we enhance the firewall, this time adding session capability, then the same communication between the laptop and server can be viewed as one.

The firewall understands that the request and the response are part of the same session and this small difference both reduces the admin overhead, so one rule instead of two, but this also lets you implement more contextual security where you can think of response traffic in the context that it's response to an original request and treat that differently than traffic in the same direction which is not a response.

Now this next point is really important.

In both cases, these firewalls don't understand anything above the layer at which they operate.

The top firewall operates layer 3 and 4, so it understands layers 1, 2, 3 and 4.

The bottom firewall does this plus layer 5.

Now what this means is that both of them can see IP addresses, ports, flags and the bottom one can do all of this and additionally it can understand sessions.

Neither of them though can understand the data which flows over the top of this.

They have no visibility into layer 7, for example, HTTP.

So they can't see headers or any of the other data that's been transferred over HTTP.

To them, the layer 7 stuff is opaque.

A cat image is the same as a dog image is the same as some malware and this is a significant limitation and it exposes the things that we're protecting to a wide range of attacks.

Now layer 7 firewalls fix many of these limitations so let's take a look at how.

Let's consider the same architecture where we have a client on the left and then a server or application on the right that we're trying to protect.

In the middle we have a layer 7 firewall and so that you'll remember it's a layer 7 firewall.

Let's add a robot, a smart robot.

With this firewall we still have the same flow of packets and segments and a layer 7 firewall can understand all of the lower layers but it adds additional capabilities.

Let's consider this example where the Categor application is connected using a HTTPS connection.

So encrypted HTTP and HTTP is the layer 7 protocol.

The first important thing to realize is that layer 7 firewalls understand various layer 7 protocols and the example we're stepping through is HTTP so they understand how that protocol transfers data, its architecture, headers, data, hosts, all of the things which happen at layer 7 or below.

It also means that it can identify normal or abnormal elements of a layer 7 connection which means it can protect against various protocol specific attacks or weaknesses.

In this example so a HTTPS connection to the Categor server the HTTPS connection would be terminated on the layer 7 firewall so while the client thinks that it's connecting to the server the HTTPS tunnel would be stripped away leaving just HTTP which it could analyze as it transits through the firewall.

So a new HTTPS connection would be created between the layer 7 firewall and the back end server so from the server and client perspective this process is occurring transparently.

The crucial part of this is that between the original HTTPS connection and the new HTTPS connection the layer 7 firewall sees an unencrypted HTTP connection so this is plain text and because the firewall understands the layer 7 protocol it can see and understand everything about this protocol stream.

Data at layer 7 can be inspected, blocked, replaced or tagged and this might be protecting against adult content, spam, off topic content or even malware.

So in this example you might be looking to protect the integrity of the Categor application.

You'll logically allow cat pictures but might be less okay with doggoes.

You might draw a line and not allow other animals sheep for example might be considered spam.

Maybe you're pretty open and inclusive and only block truly dangerous content such as malware and other exploits.

Because you can see and understand one or more application protocols you can be very granular in how you allow or block content.

You can even replace content so if adult images flow through these can be replaced with a nice kitten picture or other baby animals.

You can even block specific applications such as Facebook and even block the flow of business data leaving the organization onto services such as Dropbox.

The key thing to understand is that a layer 7 firewall keeps all of the layer 3, 4 and 5 features but can react to layer 7 elements.

This includes things like DNS names which are used, the rate of flow so how many connections per second, you can even react to content or headers.

Whatever elements are contained in that specific layer 7 protocol which the firewall understands.

Now some layer 7 firewalls only understand HTTP, some understand SMTP which is the protocol used for email delivery.

The limit is only based on what the firewall software supports.

Now that's everything that I wanted to cover at a high level.

Coming up in future videos I'm going to be covering how AWS implements layer 7 firewall capability into its product set.

For now though this high level understanding is what I wanted to help with in this video.

So go ahead and complete the video.

Thanks for watching and when you're ready I'll look forward to you joining me in the next.

Welcome back.

In this lesson, I want to introduce the AWS Secrets Manager.

It's often one that gets confused with the SSM Parameter Store.

Inside the Parameter Store, you can create secure strings which allow you to store passwords.

So logically, it's confusing.

So when should you use the Parameter Store versus the Secrets Manager?

So let's take a look at that and make sure that you're 100 percent comfortable with selecting between both of these products for the exam.

So for Secrets Manager, the functionality that the product provides and the way that it's architected, they're both pretty easy to understand.

For the exam though, the main thing to lock in is when you should use Secrets Manager versus Parameter Store.

So let's get that out of the way.

It shares functionality with Parameter Store.

So that's the starting point.

Don't worry too much if for certain scenarios, you can't really pick between them because for certain things you can use either and achieve the same result.

Secrets Manager though, as the name suggests, is designed specifically for secrets.

So this means things like passwords and API keys.

So in the exam, if you see those keywords, so API keys or passwords, then you should default to Secrets Manager.

Secrets Manager is usable from the console, the CLI, the API, and software development kits.

It's actually designed architecturally to be integrated inside other applications.

So that's one of the most common use cases that Secrets Manager is integrated with other applications.

Another key differentiator of Secrets Manager is that it actually supports the automatic rotation of secrets.

This uses Lambda.

Essentially, a Lambda function is invoked periodically and it's used to update the secrets.

And for certain AWS products such as RDS, Secrets Manager supports direct integration.

So as well as being able to periodically change a secret that's stored inside Secrets Manager, the product can also make sure that any authentication built into that product such as RDS is also changed.

So it's kept synchronized with Secrets Manager.

So if Lambda is invoked and changes a secret, so rotates a secret, then the password inside an RDS instance can also be changed.

And that integration is only supported for a certain limited set of products.

And RDS is one of those products.

So in the exam, if you see any mention of rotating secrets and more specifically, rotating secrets with RDS, then it's almost certain that Secrets Manager is the right answer.

So the product at a fundamental level, just like with Parameter Store, lets you store secrets.

But in this case, this product is specifically designed and focuses on the storage and rotation of these secrets.

So the product keeps them safe, they're encrypted at rest.

It integrates with IAM, so you can use IAM permissions to control access to the secrets.

It rotates them and clients can use Secrets Manager to access the secrets and then use these to communicate with say a database in a safe and secure way.

So let's have a look visually at an example architecture that involves Secrets Manager.

I want to use an example of a web application which allows you to share funny images showing happy things.

So you guessed it, we're talking about Categor.

Categor uses the Secrets Manager SDK.

So the SDK is part of the application and it uses this SDK to retrieve database credentials.

So the SDK uses IAM credentials for authorization, generally a role, but it might also use access keys, even though this is less ideal.

These credentials are used to interact with Secrets Manager and retrieve the secrets for the database that the application uses.

So once the application has these secrets, it can use them to securely access the database.

Now, so far all of this functionality could also be provided using the SSM parameter store.

It has the capability to store secure strings and we could use these secure strings which are encrypted to store any database connectivity information.

What sets the Secrets Manager apart is that periodically the Secrets Manager can invoke a Lambda function to rotate credentials.

Now the Lambda function will require permissions to do this and it gets those permissions from an execution role and it will use these permissions both to update the secret that is stored within Secrets Manager, but also if you use supported products such as RDS, then the actual authentication information inside RDS can also be updated and kept in sync with the secrets that are inside the Secret Manager product.

So if you're using a supported integrated product, then everything is managed end to end by Secrets Manager.

It can handle the rotation of credentials, it can handle the update of the products which use those credentials and as long as the application keeps checking in with Secrets Manager, it can always ensure that it has access to the most updated versions of those secrets.

Now it's also worth mentioning that secrets are secured using KMS, so you never risk any leakage via physical access to the AWS hardware and KMS also ensures role separation which means that you need permissions both to KMS and to Secrets Manager in order to access secrets and decrypt them.

With a specific focus on the exam, if you see any questions where you suspect that Secrets Manager might be involved, you need to do keyword analysis.

So you need to determine if the question mentions anything in the area of secrets, you need to check if the question mentions anything to do with rotation and if it mentions either of them or both of them and if it's also mentions a product such as RDS, then you're almost certain to be using Secrets Manager for the correct answer.

So the key differentiating point between Secrets Manager and the parameter store tends to be whether it explicitly mentions secrets, whether it talks about rotation and whether it talks about integration with specific products, specifically RDS.

Now most questions in the exam won't present you with a situation where you have to pick between Secrets Manager and the parameter store.

Generally the question will present a scenario and you will either have parameter store or Secrets Manager as an answer.

It's very rare that you have both.

If you do have both, then you need to be looking for keywords around rotation, integration and the specific mention of secrets.

And if you see any of those, it's likely that Secrets Manager will be the correct answer versus the parameter store.

If you need to store anything but secrets, so hierarchical configuration information, maybe the configuration for the CloudWatch agent, anything of that nature, that tends to be the type of situation where you would use the parameter store.

If it's just Secrets, if it's rotation, if it's product integration, then it's Secrets Manager.

With that being said, that's everything that I wanted to cover in this architectural theory lesson.

Go ahead, complete the video, and then when you're ready, I'll look forward to you joining me in the next.

Welcome to this video where I'm going to very briefly talk about AWS Transfer Family.

Now this is going to be an introduction video.

If the topic you're studying requires any additional information, either theory or practical, then there will be additional videos.

If not, then this is all that you need to know.

Now I want to keep this video as brief as possible.

So let's jump in and get started.

So AWS Transfer Family is a product which provides a managed file transfer service.

So it allows you to transfer files to or from S3 and EFS.

Now the reason that it's managed is that it provides managed servers which support various protocols.

So this product allows you to upload and download data to and from EFS and S3 using protocols other than those two native protocols.

So if you need to access S3 or EFS and not use S3 or EFS natively, then Transfer Family is the product which allows this.

So it allows you to interact with both of these services using a number of common protocols.

First is FTP, which is the File Transfer Protocol, and this is an unencrypted File Transfer Protocol.

This is a legacy protocol.

It's been around for decades.

It's unencrypted, and so the usage of it is relatively niche.

You've also got File Transfer Protocol Secure, or FTP-S, and this is the File Transfer Protocol, but with TLS encryption.

This is something which adds additional layers of functionality to FTP.

We've also got the Secure Shell, or SSH, File Transfer Protocol, known as SFTP.

So this is File Transfer running over the top of SSH.

And then lastly we have the Applicability Statement 2, or AS2 protocol, and this is used for transferring structured business to business data.

Now this is relatively niche, but it was added to the product because this is relatively common in certain industry sectors.

So you might use AS2 in industries where workflows with compliance requirements, so data protection and security features, need to be built into the protocol.

You might use AS2 for various supply chain logistics processes, payment workflows, or other business to business transactions, or integrations with enterprise resource planning and customer relationship, or CRM systems.

So using AS2 is something that you'll only be doing in a certain set of niche situations, and you'll know that you need this protocol.

FTP, FTP-S, and SFTP are much more common, and you'll generally find these in a wide variety of industries.

Now Transfer Family also supports a wide variety of identity providers, so the service can have built-in identities.

You can utilize the Directory Service product of AWS, or you can use custom identity providers, and this uses Lambda or API Gateway.

And then lastly, one really important feature of Transfer Family is the Managed File Transfer Workflows, or MFTW.

And you can think of this as a serverless file workflow engine.

So this can be used for when files are uploaded to the product, you can define a workflow as to what happens to that file as it gets uploaded.

So things like notification or tagging.

And this can be really effective if you need to build in the Transfer Family as part of other process-based workflows that you have within your business.

So with Transfer Family, you get access to all of these different file transfer protocols via a Transfer Family server within AWS without the need to manage any server infrastructure.

So if you need to interact with S3 or EFS using existing workflows where you can't change your applications, then you can use Transfer Family.

Now let's take a look visually at the architecture of the product.

So to high level, this is how Transfer Family works.

So we have an AWS environment and inside we have S3 and EFS.

And we want to grant access to either or both of these to an external user.

And let's say it's a medical user of some kind and they want to use the SFTP protocol.

Now to do that, we'd configure Transfer Family and with Transfer Family, we create servers which are enabled for one or more protocols.

These servers are configured to communicate with our backend storage resources using IAM roles in order to get permissions.

Now once this is configured, it means that our user can access these resources potentially using a custom DNS name using a protocol which they support and not the native S3 or EFS access methods.

And as I mentioned moments ago, Transfer Family supports a range of authentication methods including built-in identities, directory service within AWS or a custom identity store.

Now that's how the service works from a high level architecture perspective.

But there is additional information that you'll need for the real world and the AWS exams where it features.

So let's take a look at how we can connect to the service over various different networking architectures.

Within Transfer Family, you create servers which you can think of as the front end access points to your storage.

So they present the supporting backend storage, so S3 and EFS via one or more supported protocols.

So SFTP, FTPS, FTP and AS2.

Now how you're able to access these depends on how you configure the service's endpoints.

And we have three different options, Public, VPC with Internet access and VPC internal only.

With Public, the endpoint is running in the AWS public zone and so it's accessible from the public internet.

This means that there's nothing to configure.

You don't need to configure anything in the way of networking or have to worry about VPCs or any other networking components.

But it does come with some shortcomings.

First, the only supported protocol is SFTP.

Second, the endpoint has a dynamic IP which can change and this is managed by AWS.

So you should generally use DNS to access it.

And then finally, it means that you can't control who can access it using features such as network access control lists or security groups.

Next, we have the endpoint types which run in a VPC.

And much of this is shared between the two.

So both run inside a VPC.

With the VPC Internet type, you can use SFTP and FTPS endpoints and also AS2.

I keep this separate because it's a pretty niche use case right now.

And most users of this product you'll tend to find will be the usual file transfer protocols.

Now with the internal only VPC endpoint type, you can use SFTP, FTPS and FTP in addition to AS2.

And this makes sense because FTP which is the one additional protocol which is supported with VPC internal only is unencrypted.

So this is not something you'll want to be running over the public Internet.

Now both of these types can be accessed from connected business networks.

So anything that's using Direct Connect and VPNs.

So anything which has connectivity into the VPC can access this service as though it was inside the VPC.

In both cases, Transfer Family provides static IPs and both can be secured using network access control lists or security groups.

So this is a security benefit versus the public endpoint type.

The main difference is that the VPC Internet type is allocated with an elastic IP and this is static and it allows it to be accessible over the public Internet in addition to within the VPC and from corporate networks.

So this is the level of detail which you need to understand as a foundation.

If for the course that you're studying you need to know anything else then there will be additional theory or practical videos following this one.

If not, don't worry, this is everything that you'll need to understand.

Now a few final points about the AWS Transfer Family.

It is multi-AZ.

The cost is based on provisioned server per hour and data transferred through the product.

So this product does not have any upfront costs and you only build while you're utilizing the product.

With FTP and FTPS, only directory service or custom identity providers are supported.

So remember that.

And for the FTP protocol, you can only use it internally within a VPC.

This one can't be public, which means you can't use it with the public endpoint type and can't use it with the VPC Internet endpoint type.

Now with the AS2 protocol, this needs to be VPC Internet or internal only.

This cannot use the public endpoint type.

So you'd use this product if you need access to S3 or EFS using existing protocols, potentially integrating it with your existing workflows where you can't change the protocol that's used, or potentially using the managed file transfer workflow feature in order to create new workflows with the product.

Now as I mentioned at a high level, this is the knowledge that you need across any AWS exams which feature this product.

If there is any additional theory or practical knowledge that you need to know, there will be follow-up videos to this one.

But at this point, that's everything I want to cover in this video.

So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about another file system provided by FSX and that's FSX for Lustre.

Now this is a file system designed for various high-performance computing workloads.

Now it's important for the exam that you understand exactly what it provides and how it's architected.

Now we've got a lot to cover so let's jump in and take a look.

In the exam you won't need to know about Lustre in detail, it's one of those relatively niche products but you'll need to distinguish between scenarios when you might use products such as FSX for Windows versus FSX for Lustre.

Now FSX for Windows is a Windows native file system which is accessed over SMB.

It's used for Windows native environments within AWS.

FSX for Lustre is a managed implementation of the Lustre file system which is a file system designed specifically for high performance computing.

It supports Linux based instances running in AWS and as a keyword to track for the exam it also supports POSIX style permissions for file systems.

Now Lustre is designed for use cases such as machine learning, big data or financial modeling.

Anything which needs to process large amounts of data and do so with a high level of performance.

Now FSX for Lustre can scale to hundreds of gigabytes per second of throughput and it offers sub millisecond latency when accessing that storage and this is the level of performance that you'll need for high performance computing across many different clients or many different instances.

Now FSX for Lustre can be provisioned using two different deployment types.

If you have a need for the absolute best performance for short-term workloads then you can pick Scratch.

Scratch is optimized for really high-end performance but it doesn't provide much in the way of resilience or high availability.

If you need a persistent file system or high availability for your workload then you can choose the persistent option.

Now this is great for longer-term storage it offers high availability but crucially in one availability zone only.

Lustre is a single availability zone file system because it needs to deliver this really high-end performance and so the high availability provided by the persistent deployment type is high availability within one availability zone only and this also offers self-healing so if any hardware fails as part of the file system it will be automatically replaced by AWS.

So this is the deployment type to choose if you need resilience and high availability of the data running on the file system.

Now you won't need to know this level of detail for the exam but I have included a link attached to this lesson which details the differences between Scratch and persistent in detail.

I find it useful to know at least at a high level the difference between these two different deployment types.

Now FSX for Lustre as with FSX for Windows is available over a VPN or Direct Connect from on-premises locations.

Obviously you will need a pretty substantial amount of bandwidth to benefit from the Lustre performance but it is there as an option.

Now it's important for the exam that you have an understanding of how FSX for Lustre works so let's have a look at that next.

Before I cover the architecture of FSX for Lustre I want to conceptually talk about what FSX for Lustre means.

What do you actually do when you use this file system?

Well the product focuses around a managed file system which you create it's accessible from within a VPC and anything connected to that VPC via private networking.

So connectivity wise it's much like EFS or FSX for Windows in that sense in that you can access it from the VPC or anything connected to it with private networking.

Now the file system is where the data lives.

It's where it lives while it's being analyzed or processed by your applications.

When you create a file system though you can associate it with a repository and a repository in this case is an S3 bucket.

If you do this when the file system is created the objects within the S3 bucket are visible in the file system but crucially at this stage they're not actually stored within the Lustre file system.

When they're first accessed by any clients connected to the Lustre file system the data is lazy loaded into the file system from the S3 repository that first time it's loaded and then it's present within the file system.

So it's important to understand that while objects initially appear to be within the file system if you're using an S3 repository they're only actually truly present in the file system when they're first accessed so they're lazy loaded from the repository into the file system.

There isn't actually any built-in synchronization so conceptually the Lustre file system is completely separate it can just use an S3 repository as a foundation.

Now you can actually sync any changes made in the file system back to the S3 repository i.e. the S3 bucket using the HSM underscore archive command.

What I want you to understand conceptually is that the Lustre file system is completely separate it can be configured to lazy load data from S3 and to write it back but it's not automatically in sync and it's the file system the Lustre file system the separate file system where the processing of data occurs.

So now that I've covered the conceptual elements of this let's take a look at how the product is actually architected.

Before I do that there are a few key points that I want to discuss so Lustre splits data up when it's storing it to disks there are a number of different types or elements to data that are stored within the file system.

First is the metadata so this is things like file names, timestamps and permissions and this is stored on metadata targets or MSTs and Lustre file system has one of these.

Then we've got the data itself and the data is split over multiple object storage targets known as OSTs and each of these is 1.17 TIB in size and it's by splitting the data across each of these OSTs that the performance levels of Lustre are achieved.

Now the performance provided by the product is a baseline performance level based on the size of the file system and the size of the file system starts with a minimum of 1.2 TIB and then you can use increments of 2.4 TIB.

For the scratch deployment type you get a baseline performance of 200 megabytes per second per TIB of storage.

For the persistent deployment type there are actually three baseline performance levels you've got 50 megabytes per second 100 megabytes per second and 200 megabytes per second per TIB of storage and for both types you can burst up to 1300 megabytes per second per TIB of storage and this is based on a credit system so you earn credits when you're using a performance level below your baseline and you consume these credits when you burst above the baseline so it shares many of the characteristics of EBS volumes but just at a much higher scale and with more parallel architecture.

So let's look at this architecture visually.

So any FSx architecture uses a client managed VPC so something that you design and maybe you implement.

Inside this client managed VPC are some clients and these are generally Linux EC2 instances with the Lustre software installed so they can read and interact with the Lustre file system.

Then at the other side of the architecture you create a Lustre file system and optionally an S3 repository for that file system.

Now depending on the size of storage that you can figure within Lustre the product deploys a number of storage servers.

These servers handle the storage requests placed against the file system and each of them also provides an in-memory cache which allows faster access to frequently used data.

At a high level the more storage provisioned the more servers and the more aggregate throughput and IOPS that FSx for Lustre can deliver into your VPC and this performance is delivered into your VPC using a single elastic network interface.

Lustre runs from one availability zone and so you'll have one elastic network interface within your client managed VPC which is used to access the product.

Now from a performance perspective any rights to Lustre will go via the ENI and they'll be written directly to disk and so that will be dependent on the disk throughput and IO characteristics.

Likewise if data is read directly from disk then it too is based on the performance characteristics of the underlying disks.

Once data is read and then accessed again so for any frequently accessed data it can use in-memory caching and at that stage it's based on the performance characteristics of the networking connecting the clients through to the Lustre servers themselves.

So at a high level this is the architecture that the FSx for Lustre product uses.

Now let's have a look at some key points that you need to be aware of for the product.

When you're creating an FSx for Lustre file system you get to create it using one of two deployment types.

I mentioned these earlier in this lesson.

The first one is scratch and scratch is designed when you want pure performance.

You might be looking to deploy some short-term or temporary workloads and the only thing that you care about is pure performance.

So for this type of workload you can use the scratch deployment type but it's really important that you know as a solutions architect that this doesn't provide any high availability nor any form of replication.

If you have any form of hardware failure then any data that stored on that hardware is lost and isn't available to the file system.

Now this doesn't mean that other data is also at risk because any other data continues to be available as part of the Lustre file system.

It's only data affected by a failure.

But what you need to understand from a file system planning perspective is that larger file systems generally mean more servers, that means more disks and that means a higher chance of failure.

So the larger the file system the more chance of failure when you're using the scratch deployment type.

Now choosing the persistent deployment type means you do have replication but crucially only within a single availability zone.

So all the hardware and all the data is replicated within a single availability zone which protects you against hardware failure but not against the failure of a full availability zone.

So using the persistent deployment type means the product will auto heal any hardware failure and data won't be lost.

But remember this is only within one availability zone.

If an entire availability zone fails then you could have data loss because hardware is not recoverable outside of that availability zone.

Now this doesn't mean data has to be at risk because with both of these deployment types you can use the backup functionality of the product to back up that data to S3 and you can either do manual backups or automatic backups.

And automatic backups have anywhere between zero to 35 days of retention and like other AWS products zero means that automatic backups are disabled.

So this at a high level is how the FSx for Lustre product works.

It's similar to FSx for Windows and similar to EFS in terms of how it's architected.

It uses elastic network interfaces which are injected into a VPC and these can be accessed from the VPC or from any other network connected to that VPC using private networking.

Now for the exam if you see Windows mentioned or SMB mentioned then it's going to be FSx for Windows and not FSx for Lustre.

If you see any mention of Lustre, any mention of really high end performance requirements, any mention of POSIX, high performance computing, machine learning, big data or any of those type of scenarios then it's going to be FSx for Lustre.

If you see any mention of machine learning and SageMaker and you need to have access to a really high performance file system then again it could be FSx for Lustre.

With that being said that is everything that I wanted to cover within this lesson so go ahead complete lesson and then when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to cover the FSx products, specifically FSx for Windows File Server.

FSx is a shared file system product, but it handles the implementation in a very different way than say EFS, which we've covered earlier in the course.

Now FSx for Windows File Server is one of the core components of the range of services that AWS provide to support Windows environments in AWS.

For a fair amount of AWS history, its support of Windows environments was pretty bad.

It just didn't seem to be a priority.

Now this changed with FSx for Windows File Server, which provides fully managed native Windows File Servers or more specifically file shares.

You're provided with file shares as your unit of consumption.

The servers themselves are hidden.

It's similar to how RDS is architected, but instead of databases, you get file shares.

Now it's a product which is designed for integration with Windows environments.

It's a native Windows file system.

It's not an emulated file server.

It can integrate with either managed Active Directory or self-managed Active Directory and this can be running inside AWS or on-premises.

And this is a critical feature for enterprises who already have their own Active Directory provision.

Now it's a resilient and highly available system and it can be deployed in either single or multi AZ mode.

Picking between the two controls the network interfaces which are available and are used to access the product.

It uses elastic network interfaces inside the VPC.

The back end, even in single AZ mode, uses replication within that availability zone to ensure that it's resilient to hardware failure.

But if you pick multi AZ, then you get a fully multi AZ highly available solution.

Now it can also perform a full range of different types of backups and this includes some client side and AWS side features.

And I'll talk about that later in the lesson.

But from an AWS side, it can perform both automatic and on demand backups.

Now file systems that are created inside the FSx product are accessible within a VPC, but also, and this is how more complex environments are supported, they can be accessed over peering connections, over VPN connections, and even can be accessed over physical direct connects.

So if you're a large enterprise with a dedicated private link into a VPC, then you can access FSx file systems over Direct Connect.

Now in the exam when you faced with any questions which talk about shared file systems, you need to be looking to identify any Windows related keywords.

So look for things like native Windows file systems, look for things like Active Directory or Directory Service integration, and look for any of the more advanced features which I'll talk about over the remainder of this lesson.

Essentially your job in the exam needs to be to pick when to use FSx versus EFS because these are both network shared file systems that you'll find on the exam.

Generally EFS tends to be used for shared file systems for Linux EC2 instances as well as Linux on-premises servers whereas FSx is dedicated for Windows environments, so that's the main distinction between these two different services.

So let's have a look visually at how a typical implementation of FSx for Windows file server might look for an Animals for Life type organization.

So we start with a familiar architecture.

We have a VPC on the left and a corporate network on the right and these networks are connected with Direct Connect or VPN and have some on-premises staff members.

Now inside the VPC we have two availability zones A and B and in each of those availability zones we have two different private subnets.

FSx uses Active Directory for its user store and so logically we start with a directory and this can either be a managed directory delivered as a service from AWS or it can be something which is on-premises.

Now this is important FSx can integrate with both and it doesn't actually need an Active Directory service defined inside the Directory Services product.

Instead it can connect directly to Active Directory running on-premises.

This is critical to understand because it means it can integrate with a completely normal implementation of Active Directory that most large enterprises already have.

Now I already mentioned this on the previous screen but FSx can be deployed either in single AZ or multi AZ mode and in both of those it needs to be connected to some form of directory for its user store.

Now once deployed you can create a network share using FSx and this can be accessed in the normal way using the double backslash, DNS name and share notation that you'll be familiar with if you use Windows environments.

So in this example a file system ID dot animals for life dot org and then a slash cat pics and in this example cat pics is the actual share.

Now using this access path the file system can be accessed from other AWS services which use Windows based storage and an example of this is Workspaces which is a virtual desktop service similar to Citrix which is available inside AWS.

So when you deploy Workspaces into a VPC not only does it require a directory service to function but for any shared file system needs it can also use FSx.

The most important thing to remember about FSx is that it is a native Windows file system.

It supports things like DGUplication, the distributed file system or DFS which is a way Windows can group file shares together and scale out for a more managed file share structure at scale.

It supports at rest encryption using KMS and it also lets you enforce encryption in transit.

Shares are accessed using SMB protocol which is standard in Windows environments and FSx even allows for volume shadow copies which in this context are a way that users can see multiple file versions and initiate restores from the client side.

So that's really important to understand if you're utilizing an FSx share from a Windows environment you can right click on a file or folder view previous versions and initiate file level restores without having to use AWS or engage with a system administrator and that's something that's provided along with the FSx product as long as it's integrated with Windows environments you get that capability.

Now from a performance perspective FSx is highly performant.

The performance that's delivered can range from anywhere from 8 megabytes per second to 2 gigabytes per second.

It can deliver hundreds of thousands of IOPS and it delivers less than a one millisecond latency so it can scale up to whatever performance requirements your organization has.

Now for the exam you don't need to be aware of the implementation details and I'm trying inside this course to focus really on the topics and services that you need for the exam.

So when things do occur I want to teach you more information than you do require for the exam but there are a lot of topics or features of different services that you only require a high level overview of and this is one of those topics.

So what I want to do now is go through some keywords or features that you should be on the lookout when you see any exam questions that you think might be related to FSx.

Now the first one of these is VSx and this is a Windows feature that allows users to perform file and folder level restores.

So this is one of the features that's provided that's unique to FSx and it means that if you have any users of workspaces if they use files and folders on an FSx share and they right click they can view previous versions they can restore from a user-driven perspective without having to engage a system administrator.

So this might come up in the exam.

Another thing you need to be aware of is that FSx provides native Windows file systems that are accessible over SMB.

So if you see SMB mentioned in the exam it's probably going to be FSx as the default correct answer.

Remember the EFS file system uses the NFS protocol and this is only accessible from Linux EC2 instances or Linux on premises servers.

If you see any mention of SMB then you can be almost certain that it's a Windows environment question and involves FSx.

Another key feature provided by FSx is that it uses the Windows permission model so if you're used to managing permissions for folders or files on Windows file systems then you'll be used to exactly how FSx handles permissions.

So this is provided natively by the product specifically to support Windows environments in AWS.

Next is that the product supports DFS which is the distributed file system so if you see that mentioned either its full name or DFS then you know that this is going to be related to FSx.

So DFS is a way that you can natively scale out file systems inside Windows environments.

You can either group file shares together in one enterprise-wide structure or you can use DFS for replication or scaling out performance.

It's a really capable distributed file system.

Now if you see any questions which talk about the provision of a native Windows file server but where the admin overhead of running a self managed EC2 instance running something like Windows server is not ideal then you know that it's going to be FSx.

So FSx provides you with the ability to provision a native Windows file server with file shares but without the admin overhead of managing that server yourself.

And lastly the product is unique in the sense that it delivers these file shares and they can also be integrated with either directory service or your own active directory directly.

So these really important things to remember for the exam and they'll help you select between other products and FSx.

Now again I don't expect you to get many questions on FSx.

I do know at least one or two unique questions in the exam but even if it only gets you that one extra mark it can be the difference between a pass and a fail.

So try your best to remember all the key features that have explained throughout this lesson.

But at that point that is everything I wanted to cover in this theory only lesson.

So go ahead complete this video and then when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about an AWS service which you will use in the real world as a solutions architect.

And it's also one which starts to feature more and more in the exam.

Now that product is AWS Datasync.

We've got a lot to cover so let's jump in and get started.

Now AWS Datasync tends to feature in the exam currently in a very light way.

You might be lucky and not even have a question on it but I do know that it features in at least two unique questions that I'm aware of.

And so you do need to be aware of what it is, what it does and the type of situations where you might use it.

So Datasync is a data transfer service which allows you to move data into or out of AWS.

Historically many of the transfer tasks involving AWS have either been manual uploads or downloads or have used a physical device like the Snowball or Snowball Edge series of transfer devices.

While Datasync is a service which manages this process end to end.

Datasync tends to be used for workloads like data migrations into AWS or when you need to transfer data into AWS for processing and then back out again or when you need to archive data into AWS to take advantage of cost effective storage.

It can even be used as part of disaster recovery or business continuity planning.

Now as a product it's designed to work at huge scales.

Each agent and I'll introduce the concept of an agent later in this lesson but each agent can handle 10 gigabits per second of data transfer and each job and I'll also introduce the concepts of jobs within this lesson can handle 50 million files.

Now this is obviously huge scale.

Very few transfer jobs will require that level of capacity or performance but in addition to that scale it also handles the transfer of metadata such as permissions and timestamps which are both essential for complex data structure migrations.

And finally and this is a huge benefit for some scenarios.

It includes built-in data validation.

Imagine if you're transferring huge numbers of medical records or scans into AWS you need to make sure that the data as it arrives in AWS matches the original data and Datasync includes this functionality as default.

Now in terms of the key features of the product it is really scalable.

Each agent can handle 10 gigabits per second of data transfer which equates to around 100 terabytes per day and you can add additional agents assuming you have the bandwidth to support it.

You can use bandwidth limiters to avoid the saturation of internet links so reducing the customer impact of transferring the data.

The product supports incremental and scheduled transfers and it supports compression and encryption.

If you're transferring huge amounts of data and have concerns over liability issues then it also supports automatic recovery from transit errors.

It also handles integration with AWS services such as S3, EFS and FSX for Windows servers and for some services it supports service to service transfer.

So moving data from EFS to EFS inside AWS even cross region and best of all it's a pay as you use service so there is a per gigabyte cost for any data that's moved by the product.

So let's quickly look at the architecture visually to help you understand exactly how it gets used and this is going to be useful for the exam.

In this example architecture we have a corporate on-premises environment on the left and an AWS region on the right.

The business premises has an existing SAN or NAS storage device which has data on it that we want to move into AWS.

And so we install the data sync agent on the businesses on-premise VMware platform and this agent is capable of communicating to the NAS or SAN with either the NFS or SMB protocols.

So most SANs or NASs or any other storage devices will support either one or both of these protocols.

So the data sync agent is capable of integrating with nearly all local on-premises storage.

Once data sync is configured the agent communicates with the data sync endpoint running within AWS and from there it can store the data in a number of different types of locations.

Examples of this include various S3 storage classes or VPC resources such as the Elastic File System or FSX for Windows Server.

Now you can configure a schedule for the transfer so either targeting or avoiding certain time periods and if you do have any link speed performance issues you can set a bandwidth limit to throttle the rate at which data sync syncs the data between your on-premises environment and AWS.

Now for the exam it's just the architecture that you need to understand.

It won't feature at the level where you'll need to be aware of the implementation details.

So at a very high level be aware that you need to have the data sync agent installed locally within your on-premises environment.

Be aware that it communicates over NFS or SMB with on-premises storage and then it transfers that data through to AWS.

It can recover from failures, it can use schedules, it can throttle the bandwidth between on-premises and AWS and then from there it can currently store into S3 the Elastic File System or FSX for Windows Servers.

So if you see any questions in the exam which talk about the reliable transfer of large quantities of data if it needs to integrate with EFS, if it needs to integrate with FSX, if it needs to integrate with S3 and support bidirectional transfer, incremental transfer, schedule transfer then it's likely to be AWS data sync that's the right answer.

Now let's finish up by reviewing the main architectural components of the data sync product.

First we have the task and a task within data sync is essentially a job.

A job defines what is being synced, how quickly, any schedules that need to be used, any bandwidth throttling that needs to take place, but it also defines the two locations that are involved in that job.

So where is the data being copied from and where is it being copied to?

Next we have the agent and I've already talked about this.

This is the software that's used to read or write to on-premises data stores.

So it uses NFS or SMB and it's used to pull data off that store and move it into AWS or vice versa.

Lastly we've got a location and every task so every job has two locations, the from location and the to location.

Examples of valid locations are network file systems or NFS, server message block or SMB and both of these are very common corporate data transfer protocols.

You tend to use NFS with Linux or Unix systems and SMB is very popular in Windows environments.

Other valid locations include AWS storage services such as EFS, FSX and Amazon S3.

Now that's all of the information that you'll need for the exam.

I just wanted to introduce the service because as I mentioned at the start, I am aware that there are at least two questions involving this product which feature on the new version of the exam.

And I want to make sure that you go into that exam understanding the high level architecture.

So if you do see DataSync mentioned, you can at least identify whether it's an appropriate use of that technology or not.

You'll find that those questions aren't asking you to interpret different features of DataSync.

You'll be asked to select between DataSync and another product or another method of getting data into AWS.

And so in this lesson I wanted to focus on exactly when you would use DataSync.

So if you need to use an electronic method, obviously Snowball or Snowball Edge aren't appropriate.

If you need something that can transfer data in and out of AWS, if that product needs to support schedules, bandwidth throttling, automatic retries, compression and cope with huge scale transfers involving lots of different AWS and traditional file transfer protocols, then it's likely to be DataSync that's the product that you need to pick.

With that being said, that's everything that you need to know for any DataSync questions on the exam.

So go ahead, complete this video and when you're ready, I look forward to you joining me in the next.

Welcome back and in this lesson I want to cover the AWS Directory service.

This is another service which I think is often overlooked and undervalued.

It provides a managed directory, a store of users, objects and other configuration.

Now it's delivered as a managed service, it has a few versions and lots of use cases.

So we do have a lot to cover in this architecture lesson.

So let's jump in and take a look.

Before we start I want to talk about directories in general.

What are they and what do they do?

Well, directories store identity and asset related information.

So things like users, groups, computer objects, server objects and file shares.

And it holds all of these objects in a structure which is hierarchical.

So like an inverted tree.

And this is often referred to as a domain.

But regardless of its name, it's essentially an inverted tree structure that holds identity related objects.

Now multiple directories, each of which provide a tree structure, can be grouped together into what's called a forest.

Now directories are commonly used within larger corporate windows environments.

You can join devices to a directory.

So laptops, desktops and servers.

And directories provide centralized management and authentication.

So it means that you can sign in to multiple devices with the same usernames and passwords.

And it allows corporate IT staff to centrally manage all of the identity and asset information in one single data store.

One of the most common types of directory that's in use in large corporate environments is Active Directory by Microsoft.

And its full name is Microsoft Active Directory Domain Services or ADDS.

But there are alternatives.

Another common one is called SAMBA, which is an open source implementation of Active Directory.

And it's designed to provide an alternative, but only provides partial compatibility with Active Directory.

And that's something that you need to be aware of when it comes to picking the mode that directory service will be operating in.

So now let's look at directory service specifically.

So directory service is an AWS implementation of a directory.

It's the same equivalent to Active Directory as RDS is to databases.

Using it means you have no admin overhead of running your own directory service.

And that admin overhead is often significant.

Directory service runs within a VPC.

It's a private service.

So to use it for services, those services either need to be within that VPC or you need to configure private connectivity to that VPC.

It provides high availability and it does this by deploying into multiple subnets in multiple availability zones within AWS.

Now there are certain AWS services such as EC2, which can optionally use a directory.

So Windows EC2 instances can be configured to join the directory and then you can use identities inside that directory to log in to that EC2 instance.

And you can also configure a directory for centralized management to various different Windows features that are running on Windows EC2 instances.

Certain services that run inside AWS require a directory.

And an example of this is Amazon Workspaces, which is a virtual desktop product where you can get a virtual operating system upon which you can run applications.

If you've ever used Citrix or anything similar, then Amazon Workspaces is essentially AWS's version of this.

But it needs a directory and it needs to be a directory that's registered with AWS.

And so it needs the directory service, the directory service product.

And to join EC2 instances to a domain via the AWS tools, you also need to have a registered directory inside AWS.

So the directory service product is this implementation.

It's an AWS supported and registered directory service within AWS that other AWS products can utilize for identity and management purposes.

Now, when you create a directory, you're going to be doing so with a number of different architectures in mind.

So it might be an isolated directory, meaning inside AWS only and independent of any other directory that you might have, or you might not have any other directory.

It can be integrated with an existing on-premises directory, so almost like a partner directory, or you can use the directory service in what's called connector mode, where it just proxies connections back to your on-premises system.

Essentially, this allows you to use your existing on-premises directory with AWS services which require a registered directory service.

It's essentially just a proxy.

Now, I want to quickly step through each of these different architectures visually before we finish up, because for the exam, you really only need to have an awareness of the architecture.

And I always find that by looking at it visually, it helps keep it in your memory for when you sit the exam.

So let's do that next.

Let's step through each of the different modes that the directory service can run in.

Now, first, we're going to look at the directory service running in simple AD mode.

This is the cheapest and the simplest way that the product can run inside a VPC.

So we start off with a VPC, and let's say that we want to run Amazon Workspaces within this VPC.

And these Workspaces will be utilized by some Animals for Life users.

So Workspaces as a product requires a directory service.

So when you log into a workspace, you're not logging into a local user, you're logging in using a user of that directory.

And so it needs some type of directory that's registered within AWS.

And one option is that we can deploy the directory service in simple AD mode.

So simple AD is an open source directory that's based on SAMBA4.

So it's a directory that aims to provide as much compatibility with Microsoft Active Directory as possible, but do it in a lightweight way.

So if you see any mention of open source or SAMBA4 or SAMBA, then think simple AD.

So you can create users and other objects within simple AD, and this can be integrated with Workspaces.

So simple AD can operate in two different sizes, and it can support up to 500 users when using the small mode and up to 5,000 users for large-sized simple AD.

And it can integrate with lots of different AWS services.

Things like Amazon Chime, Amazon Connect, Amazon QuickSite, Amazon RDS, WorkDocs, WorkMail, WorkSpaces, and even the AWS Management Console allows you to sign in with users of the directory service.

But there are other things such as EC2, which can utilize the directory service as well, either from the console or by manually configuring the operating system of the EC2 instance.

When you deploy a simple AD directory service, you're actually deploying a highly available version of SAMBA, and so anything that can join this SAMBA directory is capable of joining the directory service directory running in simple AD mode.

Now, the critical thing to understand about simple AD mode is that it's designed to be used in isolation.

It's not designed to integrate with on-premises systems, and it isn't a full implementation of something like Microsoft Active Directory.

If you need something bigger and more feature-rich, then you can choose to use Managed Microsoft AD mode.

And this mode is designed for when you want to have a direct presence inside AWS, but also when you have an existing on-premises environment.

Using this mode, you can create an instance of the directory service inside AWS, and architecturally, it's similar to simple AD.

You can have users which are created within the directory service itself hosted inside AWS, and once created, services inside AWS can integrate directly with the directory service.

But in addition, and this is a set of features that goes beyond simple AD mode, you can create a trust relationship with your existing on-premises directory, and this needs to occur over private networking, so either a direct connect or a VPN connection.

Now, the benefit of this mode is that the primary location is in AWS, and it trusts your on-premises directory.

It means if the VPN fails, the AWS services which rely on the directory will still be able to function.

So when you deploy directory service in Microsoft AD mode, it means that it's a fully fledged directory service in its own right.

It's not reliant on any on-premises infrastructure to function.

But also, it's an actual implementation of the Microsoft Active Directory, specifically the 2012 R2 version, and so it directly supports applications which require Microsoft AD features, such as schemers and schema extensions.

And these products include things like Microsoft SharePoint, as well as Microsoft SQL Server-based applications.

So if you see questions in the exam about requiring an actual implementation of Microsoft Active Directory, complete with trust relationships with an on-premise Microsoft Active Directory, then it's the managed Microsoft Active Directory mode that you need to use, not simple AD.

Now, there's one final mode that you need to be aware of, and I have seen this come up in the exam, and this mode is AD Connector.

Consider a scenario where you only want to use one specific AWS service that has a requirement of a directory service, let's say Amazon Workspaces.

And in this example, you already have an on-premises directory, and you don't want to create a brand new one just to use this one single product.

Well, AD Connector offers a solution.

To utilize AD Connector, you would need to establish private network connectivity between your AWS account and your on-premises network.

So an example of this could be a VPN.

And then once this VPN is established, you would create the AD Connector and point the AD Connector back at your on-premises directory.

Now, what's critical to understand about the AD Connector is that it is only a proxy.

It just exists as an entity to integrate with AWS services.

So any AWS services which need a directory, they will see the AD Connector, and they'll know they'll have access to an active directory instance, but they won't care where.

And it's the AD Connector itself that's pointing back at your on-premises directory.

It doesn't provide any authentication of its own.

It is simply a proxy.

It simply proxies the requests back to your on-premises environment.

So using this architecture, the primary directory is still located on-premises, and all requests for authentication from services are proxied all the way back to your on-premises directory.

Now, this means that you don't need to provision an additional directory service specifically to use AWS products and services, but it does mean that if the private network connectivity fails, then the AD Connector stops working, and any services which use it could experience problems.

So you would use this option only when you already have a directory on-premises and just want to use AWS products and services which need a directory without deploying a brand new one.

Now, one of the important things for the exam is to know when to pick between the different modes for directory service.

You should always start off with simple AD.

Simple AD is your default.

It's designed for simple requirements.

If you need an isolated directory within AWS, if you don't need any connectivity with on-premises, you simply need to use it to support any AWS products and services which require a directory than use simple AD mode.

You can move to using Microsoft AD if you have any applications in AWS which need an actual implementation of Microsoft Active Directory.

If you have any products which expect an actual implementation of Microsoft Active Directory, you need to use this mode, or if you need to have that trust relationship with your existing on-premises Microsoft Active Directory, that's another reason to use the Microsoft AD mode.

It's critical to understand that this is actually an implementation of Microsoft Active Directory.

It's not emulated in any way.

It's not been adjusted by AWS.

It's essentially a managed deployment of a Microsoft Active Directory set of domain servers.

Now finally, if your only requirement is to use AWS services which need a directory, but you don't want to store any directory information in the cloud, you don't want to manage a directory in the cloud, you simply want the ability to use the products and services that require a directory.

That's when you would use the AD connector.

The important thing to remember about the AD connector, it doesn't provide functionality of its own.

You create it, you have to point it back at your on-premises environment, and it requires connectivity to your on-premises environment and for that on-premises environment to be fully functional.

If either of those two things are not the case, then the AD connector will fail.

Now one of the major differences between the older SAAC01 exam and the newer C02 exam is that there are a lot more Windows environment-based questions.

And that's the reason I've included this lesson on the directory service.

You need to be aware of all the different products and services that can be used to support and implement Windows environments within AWS and enable hybrid mode operation with any Windows environments that you have on-premise.

Now the questions at an associate level won't be very challenging.

They'll focus purely on this high-level architecture.

So you won't need to be aware of the implementation details for the exam.

And that's why I've chosen to make this a theory-only lesson.

So this lesson provides all the information that you'll need to answer any directory service questions that you might get in the exam.

So at this point, that's everything I wanted to cover.

So go ahead, complete this video.

And when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about a really effective set of services which AWS provide for moving data between on-premises and AWS.

So these services are Snowball, Snowball Edge and the AWS Snowmobile.

Now we've got a lot to cover so let's jump in and get started.

At a high level the Snowball series, so Snowball, Snowball Edge and Snowmobile are designed to move large amounts of data in or out of AWS.

In many cases, especially with smaller projects, you can just upload data over an internet connection or in some cases if you have specific requirements you can use Direct Connect.

But there are situations where the amount of data that you need to move makes this impractical.

Either because of the speed of your internet connection making a transfer of this size impractical or because you need to get huge amounts of data into AWS as quickly as possible.

Now the devices in this product series are physical storage units either suitcase sized or the size of a truck literally carried on trucks inside a custom built shipping container.

Now you can either order them empty, load them up with data and then return them to AWS or the reverse so you can order them with data, receive them, empty off the data and return the appliances to AWS.

Now the key things that you need to know for the associate exam is not specifically the product specifics but rather when and where you need to use the products.

So you don't need to be aware of the implementation details just the architecture.

So that's what I want to cover in this lesson and we're going to start with Snowball.

Now when you're using the Snowball product it's actually a physical process you're interacting physically with AWS and so it's a device that you order from AWS you log a job and the device is delivered to you so it's not an instant process.

Any data which is stored on a Snowball is encrypted using KMS so there is encryption at rest.

Anything that's persistently stored on that device is encrypted using KMS.

Now there are two types of Snowball devices.

You can get a 50 TB device and an 80 TB device.

Now in terms of network connectivity you can connect to the Snowball using either 1 gig or 10 gig networking so that's important to make sure that you have that physical, cabled network connectivity wherever you get the Snowball delivered to because you will need to use physical networking.

Now whilst the capacity of the normal Snowball device is either 50 TB or 80 TB the economical range for using Snowball is generally within the region of 10 TB to 10 PB of data so if the amount of data that you need to transfer in or out of AWS is in that range then it tends to be economical to use Snowball rather than transferring the data across the internet or across Direct Connect or any other connection technology.

For large amounts of data physical transfer using Snowball is often the most economical.

Now one of the benefits of using Snowball or Snowball Edge that I'll cover next is that you can order multiple devices so whilst it's got a 50 or an 80 TB capacity whilst the economical range is 10 TB to 10 PB you can order multiple devices and get them sent to multiple business premises so that is a really important architectural benefit you could order 10 Snowballs and have one deployed into each of 10 business premises and use those devices to ingest data send it back to AWS and get that data accessible inside your AWS environment so remember for the exam multiple devices multiple premises and try your best to remember the economical range so 10 TB to 10 PB.

Now Snowball as a device only includes storage so it's only a storage device it doesn't include any compute capability and that's important because that's in contrast to the next product that I want to talk about which is Snowball Edge.

So Snowball Edge comes with both storage and compute so it's like the Snowball product but in addition it comes with compute capability so architecturally you tend to use Snowball when you've got large amounts of data to ingest or get out of AWS with Snowball Edge you've got some other architectural patterns that you can use that involve compute and we'll talk about that in a second but another benefit of Snowball Edge is that it has a larger capacity versus the Snowball product and it also offers faster networking so you've got 10 gig over RJ45 10 or 25 over SFP and then 45 50 or 100 over QSFP plus so whatever local connection technology you have you can generally achieve faster connection to Snowball Edge versus Snowball.

Now there are actually three different types of Snowball Edge first we've got storage optimized but there's also a slight variant of the storage optimized series which includes EC2 capability so by default the storage optimized version of Snowball Edge comes with 80 TB of storage, 24 vCPUs and 32 GIB of memory but if you include the EC2 capability option it also comes with 1 terabyte of local SSD so you can actually run EC2 instances on top of this using that 1 terabyte of SSD.

Now we've also got compute optimized variants which include 100 TB of storage plus 7.68 gig of NVME storage so this is storage which is directly attached to the PCI bus it's super fast super low latency and that's beneficial when you've got aggressive compute requirements that you want to run on the Snowball Edge.

The compute optimized also includes 52 virtual CPUs of capacity and 208 GIB of memory and then finally there's also the compute optimized with GPU capability so in addition to all of those above resources it also includes GPU capability so for any scientific analysis any modeling or any parallel activities that benefit from a GPU then you can also get the Snowball Edge with GPU capability.

Now in terms of when you'd use the Snowball Edge versus the Snowball, Snowball is older generation it's purely for storage Snowball Edge includes compute so if you've got any remote sites where you need to perform data processing on data as it's ingested then you should use the Snowball Edge.

If you've got higher capacity requirements all you need faster networking if you do really have lots of data and you need to make sure that you load it onto the device as quickly as possible so that you can ship that device back to AWS then by having the faster networking provided by the Snowball Edge the turnaround time can be quicker.

Now the last piece of this product set is the Snowmobile and the Snowmobile is a portable data center within a shipping container on a truck.

Now I literally mean this it is a truck that's delivered to your business premises on the back of this truck is a custom design shipping container and in that shipping container is a portable data center so this is a product that needs to be specially ordered from AWS.

It's not something that's available in high volume and it's not available everywhere it's something that is generally only used when you have a single location with huge amounts of data that you need to ingest into AWS.

Think about situations where you've got a large enterprise who have a huge amount of data or maybe you're doing a traditional data center migration of anything over 10 petabytes of data that's the type of scenario where you might use a snowmobile.

Now snowmobiles can actually store up to a hundred petabytes of data per snowmobile.

Now the process is that you order a snowmobile it's driven to your data center location the back of the shipping container is opened up and they expect to connect into data center grade power and data networking so essentially this is driven to your location and you physically plug it into your data center and use it to transfer data into.

Now for the exam remember it's a single truck.

Why this matters is that it is not economical for smaller than 10 petabytes or when you have multi site migrations.

You have to remember that this is one single device so if you have for example 10 petabytes of data but that spread across lots of different sites then logically you would be using the snowball edge product.

If you've got all this data on one single site then potentially it's more economical to use the snowmobile.

You can't physically split a snowmobile into multiple trucks you can't do a road trip around your different data centers a snowmobile drives out to one location it's plugged in it's used to transfer data into and then it drives off back to AWS.

So if your requirement is multi site then unless it really is a huge scale migration you would not look to use a snowmobile.

So these are the three different products the snowball the snowball edge and the snowmobile and it really will benefit you in the exam to understand the exact set of requirements when you would and wouldn't use each of these products.

I don't expect you to get any questions which test your knowledge in detail you certainly won't get any questions where you need to physically know what the process of ordering and the process of transferring but I think from an architectural perspective you will get massive benefit from knowing the details on each of these products and that's what I've tried to do in this lesson.

With that being said that is everything I wanted to cover in this lesson so go ahead and complete the video and then when you're ready I look forward to you joining me in the next.

Welcome back and in this final part of the Storage Gateway series I want to talk about storage gateway running in file mode.

So far I've covered volume mode which is where Storage Gateway handles raw block volumes and VTL mode or tape mode where Storage Gateway pretends to be a physical tape backup system.

Running in file mode as you can guess from the name Storage Gateway Managers Files.

Now we have a lot to cover because this is one of the most feature-rich modes of Storage Gateway so let's jump in and get started.

Now I want to stress one thing right at the start for any storage gateway questions in the exam if you see volumes mentioned then you should default to volume gateway.

If you see tapes mentioned default to VTL mode if you see files mentioned then default to file mode and only move away if you see something that eliminates any of those options.

Now File Gateway bridges on-premises file storage and S3.

It links local file storage and an S3 bucket.

With a file gateway you create one or more mount points or shares and these are available via two protocols either NFS which is generally used for Linux servers and workstations or SMB which is a Windows Network file sharing protocol.

Now these are another pair of keywords which will help you distinguish between volume gateway, tape gateway and file gateway.

So if you see any mention of NFS or SMB with a storage gateway question that concerns files then you know it's going to be the file gateway.

Now these file shares or mount points that you create within the file gateway map directly to one S3 bucket which is in your account.

Now you manage this S3 bucket and you have visibility of this S3 bucket.

This means that when you store files onto a mount point over SMB or NFS they appear in the S3 bucket as objects.

If you store objects into an S3 bucket they're visible on the corresponding mount point on-premises.

Now this is essentially the key benefit of using storage gateway running in file mode.

It translates between on-premises files and AWS based S3 objects and this is super powerful from an architecture perspective.

Like the other storage gateways it typically runs on premises and to ensure performance it also does read and write caching and that's to ensure the performance that you achieve is more land-like so you get the same or similar level of performance as anything else which runs on a local area network.

Now the file gateway isn't an overly complex product.

Essentially what you see on screen is what it does but where the power comes from is how it integrates with S3 and how you can take advantage of the S3 features to implement some really useful architectures.

Now over the remainder of this lesson I want to step through those architectures so that you can get some idea of how you can use it effectively in production and answer any questions relating to the file gateway that you might experience in the exam.

Now a typical architecture with file gateway starts with a business premises like the one on the left and AWS on the right.

File gateway runs as a virtual appliance in most cases and it has local storage that it uses as a read and write cache and this gives whatever data is managed by the storage gateway near local area network performance.

Now on the storage gateway end so on-premises we create file shares and each of these file shares are linked with a single S3 bucket running in your account.

Now this link between a file share and an S3 bucket creates what's known as a bucket share.

This is one S3 bucket and one linked file share.

These file shares can be accessed from any local servers using NFS in the case of Linux servers and SMB for any Windows servers and if you're using a Windows share you can also use Active Directory authentication for even better integration with a Windows environment.

Now the reason why file gateway is so awesome is because the file shares and the buckets are linked together.

It means that S3 objects are visible in the file share and vice-versa.

So files that are on-premises map directly onto objects running in AWS so there's a mapping between the file name and the object name and the structure of on-premise file shares is preserved by building that structure into the object name within S3 and this is much like how S3 emulates a nested file system structure within what is a flat object storage system by building it into the object name.

So in this case a file on the left called winky.jpeg will be represented by an object called winky.jpeg inside the S3 bucket.

Another file called ruffle.png inside the omg wow folder will be represented by an object called forward slash omg wow forward slash ruffle.png so you can see how a structure is emulated by building that in to the object name.

Now you can have 10 of these shares per storage gateway and crucially the primary data is held on S3 so the only thing that stored locally is within the local cache and this holds data written or read from the storage gateway and is designed to improve performance to near those achieved from any other resources running on a local area network.

Now because the objects are stored within S3 you have the ability to integrate with other AWS services.

You can use S3 events and Lambda as well as other AWS services such as Athena.

Anything which can use S3 as a source location will have access to any files that you store on S3 indirectly using the file gateway.

So at a high level this is the architecture of a file gateway.

It allows you to extend your local file storage into AWS using S3 so if you see the keyword file in the exam maybe with the keyword extension then a possible answer is going to be the storage gateway running in file mode but the product goes far far beyond this.

The reason it's my favorite mode of storage gateway is because it enables some really cool hybrid architectures.

Now the architecture that's on screen now is a simple two-site hybrid architecture with on-premises on the left and then AWS on the right architecture.

In this architecture we still have the storage gateway in the top on-premises environment on the left of the screen and there's still the one-to-one relationship between the files presented by the storage gateway and the object in the S3 bucket but we can add another on-premises environment at the bottom and this environment also presents the same set of objects from the same bucket and it presents those as files.

Now there are some concerns to keep in mind.

First when you update a file on a local storage gateway that update is copied into S3 that's automatic but to save on resource usage and avoid unnecessary S3 listings there's no automatic version of that in reverse.

When you list the file share on-premises you're listing the most recent copy of the S3 bucket that the gateway is aware of.

In this example if you added a new cat picture to the top storage gateway that would create a new object immediately in the S3 bucket but if you then listed the file share in the bottom on-premises environment it wouldn't show until you initiated a listing.

Now there's a feature of storage gateway called notify when uploaded and I'll make sure I put a link attached to the lesson which details this functionality but at a high level this sends an event using cloud watch events which can inform the other storage gateways when an update has occurred but this needs to be designed into your solution it doesn't occur by default and another point to be aware of is that file gateway doesn't support any form of object locking this means that if two users are editing the winky.jpg file in the top and bottom environments and they both write there isn't any form of checking or control over this access and this can result in data loss where one update overwrites another so either make sure that one of the shares is read and write and everything else is read or implement some form of control on who accesses files and when.

Another powerful architecture which is supported when running storage gateway in file gateway mode is replication.

Given this architecture where we have two customer sites linked to the S3 bucket in US East 1 we could create another bucket in say US West 1 and then configure cross region replication of that data between those two buckets this gives us a nice way to implement multi-region DR without any significant changes to infrastructure or much in the way of additional costs.

We can also use file gateway and S3 lifecycle management together.

Let's say that we had this architecture so we have on-premises on the left AWS on the right using file gateway means that the files and objects remain in sync because it's using S3 there are different storage classes available for objects and three examples are S3 standard S3 standard infrequent access and S3 Glacier.

When you create a file share you specify the default storage class to use and this is generally S3 standard but over the top of this default you can create life cycle policies which run within an S3 bucket maybe you configure them to move objects from standard to standard IA after 30 days and behind the scenes objects are moved automatically between these two different classes and then because this is an automatic process maybe it repeats as additional objects reach a certain age more objects are moving between these different storage classes now multiple steps are allowed so in addition to the move after 30 days to standard IA you could also have a 90-day move to Glacier meaning that objects over time move towards cheaper storage this process happens in the background automatically which means it's cost-effective and because the primary copy of all data is in S3 any on-premises locations automatically benefit from this cost-effective storage system.

File Gateway is a really cool product but to fully appreciate it it helps to experience it in practice but for the exam this lesson has covered everything that you'll need I've covered all of the main features and architectural patterns of file gateway for the exam try to make sure that you're familiar with when you'd use each type of storage gateway so when it makes sense to use volume stored or when it makes sense to use volume cached what situations is VTL mode useful and then the same question for file gateway now if you're in doubt come and talk to me on techstudieslack.com and we can discuss this in more detail but with that being said thanks for watching go ahead complete the video and when you're ready I look forward to you joining me in the next.

Welcome back and in this lesson of the Storage Gateway series I wanted to cover Storage Gateway running in Tape mode, also known as VTL mode, which stands for Virtual Tape Library.

Now try your best as you go through this lesson to ignore the fact that the logo looks like a toilet roll.

It's one of those things where once you've seen it it can't easily be unseen.

I also often forget that there might be some of my students who don't actually know what Tape backup is, so I want to quickly cover that before I show you how Storage Gateway can integrate Tape backups and AWS.

So let's jump in and take a look.

Large enterprise backups in recent times tend to run in one of three ways.

Backup to Tape, backup to disk or off-site backup to a remote facility over a network link.

For this lesson we're focusing on Tape backup.

Now there are various different types of tapes and one popular type is called LTO which stands for Linear Tape Open.

This is an example of some LTO tapes.

They come in various different generations and one of the more recent is LTO 9 which is capable of storing 24 terabytes of data per tape and this is uncompressed data.

If you add compression it allows for storage up to 60 terabytes per tape.

All Tape backup solutions have at least one Tape drive.

A Tape drive can logically be empty or it can have a tape inserted.

Assuming it does have a tape inserted then the Tape drive can read from the tape or write to the tape and it's sequential meaning not random access like disk or SSD based storage.

Now to find data you need to seek through the entire tape, locate the data and then read it and data that's stored on tape can't easily be updated.

You need to overwrite the data that's already on that tape.

It's not really possible to modify data stored on tape.

Essentially it's designed as a medium which allows write as a whole or read as a whole.

Now as well as a tape drive you have what's known as a Tape loader sometimes called a Tape robot.

Think of this as a literal robot arm which can insert tapes, remove tapes or swap tapes between a drive and somewhere else.

Now a tape library is a piece of equipment which often fits in a rack or it's the size of one or more racks.

It contains one or more tape drives, one or more tape loaders so the robots which move tapes and a collection of slots and these slots can store tapes when they're not in the drive.

So a tape library could have room for eight tapes, 32 tapes, hundreds of tapes or even thousands of tapes.

So when we're discussing tape backup we've got a number of different components.

We've got the drive itself where tapes go to be read from or written to.

Next we've got the library and the library consists of a tape drive or tape drives, the robots and then a number of slots that can store tapes when they're not in the drives themselves and then thirdly we've got a tape shelf.

Now this is a throwback to the physical world where tapes were stored on shelves.

If you see a tape shelf mentioned it's anywhere which isn't the library so another location which could be in the same building or a different physical site entirely.

In a traditional tape backup situation this is what the architecture might look like so on the left we've got a business premises and inside it we've got a number of application and data storage servers which aren't shown together with a backup server and a tape library.

Now the backup server connects to the tape library using a protocol known as iSCSI and this iSCSI connection exposes a few devices.

The most important ones being one or more tape drives and a media changer and a media changer is just another name for a tape loader or a tape robot.

Now the important thing to realize is that everything has a cost so the equipment cost money to buy that's the tapes the software and the library and their capex costs and then to operate it there's ongoing costs so licensing maintenance and the staffing costs it's not cheap to run an enterprise grade backup architecture such as the one that's on screen now.

In addition to this though we have the other side of the architecture off-site tape storage which is often managed by a third party.

This location is often a decent distance away from your primary business premises and this is done to provide resilience in the event of a disaster.

Now only tapes which are in the library itself can be used for backups and to keep data safe any tapes which aren't being actively used are moved from your primary premises to off-site tape storage and this transport costs money and takes time so we have a few main problems with this architecture we have the cost to purchase the cost to maintain the cost to operate and the time and cost to move things around between our primary premises and our off-site tape storage and storage gateway running in VTL or tape mode fixes many of these problems so let's take a look at how.

Using storage gateway and VTL mode much of the architecture changes about the only thing which is shared is that we still have a business premises and a backup server.

Instead of the on-premises tape library we use storage gateway in tape or VTL mode and this presents itself in the same way to the backup server using iSCSI.

Now this is actually part of the design the backup server sees this as a normal tape loader it doesn't know the difference between this and the previous physical tape library.

There are very little if any software changes required beyond connecting the backup software to the storage gateway.

The storage gateway presents a media changer and the tape drives but it looks like a normal physical tape library but it has an upload buffer and a local cache much like volume gateway running in cache mode.

It uses these to store data which is being actively used essentially virtual tapes and I'll talk about these in a second.

Instead of using physical tapes which are present in a local tape library the storage gateway communicates with the storage gateway endpoint within AWS so this is much like the volume gateway.

The storage gateway endpoint then presents two main capabilities the virtual tape library or VTL which is backed by S3 and the virtual tape shelf or VTS which is backed by either Glacier or Glacier Deep Archive.

Now conceptually the virtual tape library is an AWS hosted version of a tape library which the storage gateway uses and the virtual tape shelf is for virtual tapes which have been logically moved out of that virtual tape library.

So the on premises storage gateway communicates with the virtual tape library it caches stuff locally and it uploads in the background to the virtual tape library.

So backups occur at LAN speeds to the local storage that the storage gateway uses and then in the background any backup data gets uploaded to the storage gateway endpoint running inside AWS specifically the virtual tape library.

Now a virtual tape can be anywhere from 100 gig to 5 terabytes.

If you have a good memory you might notice that the 5 terabyte maximum size for a virtual tape is also the maximum size of an S3 object and that's because these virtual tapes are stored using S3 in an AWS managed storage gateway bucket.

Now the storage gateway can handle a total of one PB of data across 1500 virtual tapes within the virtual tape library.

Remember this is the part which uses S3.

When virtual tapes aren't being used for anything they can be exported within the backup software and this marks them as not being within the library.

Now in the physical architecture which I showed you on the previous screen this would then mean ejecting them from the physical tape library and moving them to off-site storage.

An exported tape simply means that it's not in the library.

Now when you export a virtual tape using storage gateway in VTL mode it archives it from the virtual tape library or VTL into the virtual tape shelf or VTS and this moves the data from S3 into Glacier or Glacier Deep Archive and that offers unlimited storage.

The limit of this storage gateway appliance is only for tapes that are stored in the virtual tape library.

Anything which you archive into the virtual tape shelf benefits from unlimited amounts of capacity.

Now Glacier is generally used for archival when there's an expectation that at some point you will need access to data so anything that's infrequently accessed can be archived into the virtual tape shelf using Glacier.

Glacier Deep Archive is used for longer term data retention where you might never need to access the data again but you do need to keep it maybe for legal reasons.

In either case if you need to access the data again then it can be retrieved from the virtual tape shelf into the virtual tape library and then it can be accessed again by the storage gateway.

Now I'm hoping by this point that you understand what this mode of the storage gateway provides.

It essentially pretends to be an iScusy tape library, an iScusy changer and an iScusy drive and it uses a combination of S3 and Glacier and Glacier Deep Archive to support the backup and restore functionality of a physical tape library.

Now this gives you a few interesting capabilities.

You can use it to maintain your existing backup system which you might need to keep but replace much of the expensive parts with economical AWS storage.

It also allows you to extend an additional backup system by using capacity within AWS so you can use this together with existing backup software and extend any limited on-premises capacity into AWS by using S3 and Glacier.

Or it also lets you do a migration so let's say that you're migrating everything into AWS but you need to maintain a historical set of tape backups.

Well you can migrate those tape backups onto storage gateway using VTL mode, decommission the physical tape hardware and then even run the storage gateway appliance and the backup server from within AWS for any data retrieval needs.

So this type of storage gateway presents some really interesting architectural possibilities.

For the exam anything which involves traditional tape backup architecture then storage gateway running in VTL mode is likely to be the correct answer.

VTL mode is a really powerful product which I've used a few times in real-world projects generally as part of data center extensions or migrations of backup platforms.

But at this point that's everything that I wanted to cover about storage gateway running in VTL mode so thanks for watching go ahead and complete the video and I'll look forward to you joining me in the next part of this storage gateway series where I'll be covering the file gateway which is probably my favorite storage gateway mode.

Welcome back.

Over the next few lessons I'm going to be covering storage gateway in more depth, focusing on the type of architectures which it can be used to support.

Now the key to exam success when it comes to storage gateway is to understand when you would use each of the modes because each of them has their own specific situation when it should and shouldn't be used.

In this lesson I'll be starting off with the storage gateway running in volume stored mode and volume cached mode so let's jump in and get started.

Storage gateway normally runs as a virtual machine on-premises although it can be ordered as a hardware appliance but it's much more common to use the virtual machine version of this product.

Now storage gateway acts as a bridge between storage that exists on-premises or in a data center and AWS so locally it presents storage using iSCSI which is a SAN and NAS protocol, NFS which is often used by Linux environments to share storage over the network and between servers and finally SMB which is used within Windows environments.

At the AWS side it integrates with EBS, S3 and the various different types of Glacier.

Now as a product storage gateway is used for things like migrations from on-premises to AWS, extensions of a data center into AWS, maybe if you're running low on storage and can't expand it anymore then you can use AWS storage instead.

You can use storage gateway to implement storage tiering, it can help with DR and it can help replace legacy tape media backup solutions.

For the exam you need to be able to pick the correct type of storage gateway for a given scenario and that's what I want to help you with within this set of lessons.

Now as a quick visual refresher a storage gateway is generally something which has run as a virtual appliance on-premises.

Architecturally you might also have some network attached storage known as a NAS or a storage area network known as a SAN also running on-premises and the storage hardware will be used by a collection of servers also you guessed it running on-premises.

Now these servers probably also have their own local disks but for primary storage they're likely to connect to either the SAN or NAS equipment.

These storage systems so SANs or NASs generally use the iSCSI protocol which presents raw block storage over the network as block devices to these servers so the servers will see them as just another type of storage device which they can use to create a file system on and then use it in the normal way.

Now this is a traditional architecture that you'll see in many businesses and what's also pretty common is that businesses specifically smaller ones have little if any funds for backups or effective disaster recovery and are often looking at AWS as a solution to rising operational costs or as an alternative to running their own data center facilities.

So how does storage gateway work?

Well volume gateway works in two different modes we've got cached mode and stored mode and they're both very different and offer very different advantages.

First let's look at stored mode.

In this mode the virtual appliance presents volumes over iSCSI to servers running on-premises and this is just like the NAS or SAN hardware.

These volumes look exactly the same as the ones presented by the NAS or the SAN and servers can create file systems on top of these volumes just as they can with the volumes presented by the NAS or SAN devices.

Now in gateway stored mode these volumes consume capacity on-premises so the storage gateway has local storage and this local storage is the primary storage location for all of the volumes that storage gateway presents out over iSCSI and this is a critical point to remember for the exam when you're using storage gateway in volume stored mode everything is stored locally so all of the volumes presented to servers are stored on local storage on-premises.

Now storage gateway in this mode also has a separate area of storage called the upload buffer.

Any data which has written to any of the volumes in the local storage is also written to this buffer temporarily and then it's copied asynchronously into AWS and this is via the storage gateway endpoint which is a public endpoint and so this can occur over your normal internet connection or a public VIF running on a direct connect.

Now this data is being copied into S3 in the form of EBS snapshots so conceptually these snapshots are snapshots of the volumes running on-premises and this occurs constantly in the background and requires no human intervention so that's the architecture of storage gateway running in volume stored mode so think about the architecture and what it enables because this is what's important for the exam.

First it's great for doing full disk backups of servers because you're using raw volumes on the on-premises side and because you're backing those up asynchronously in the form of EBS snapshots then you have a great full disk backup regime happening and this offers excellent RPO and RTO values because these snapshots can be quickly restored without much in the way of lost data.

Volume gateway in stored mode is also great for disaster recovery because the EBS snapshots which are created can themselves be used to create new EBS volumes.

In theory you can provision a full copy of an on-premises server within AWS by just using these EBS snapshots but and this is probably the most important thing to remember for the exam what this mode doesn't allow is extending your data center capacity because the primary location for data using this mode of storage gateway is on-premises.

For every volume that storage gateway presents you have a full copy of that data on your local on-premises storage.

If you have capacity issues this mode won't do anything to help so for the exam if you're dealing with volumes and you need something to improve capacity at an on-premises or data center location then this is not the mode to help you with that.

If you need to keep low latency access to data then this mode works because the primary location of that data is on-premises.

If you need help with full disk backups or disaster recovery then this mode potentially is ideal.

Now I stress full disk because in the next few lessons in this series I'll be covering other modes of storage gateway which also help with backups.

Volume gateway deals in volumes raw disks presented over iSCSI.

Now there are some key facts which are worth knowing you won't need to remember these figures exactly for the exam but for volume stored mode you can have 32 volumes per gateway 16 TB per volume and that represents 512 TB per gateway in volume stored mode so that's volume gateway in stored mode and next I want to look at volume gateway in cached mode.

Cached mode is useful for a different set of scenarios.

Gateway cached mode shares the same basic architecture you still have the storage gateway running as a virtual appliance or in some cases physical you still have the local servers and they're still being presented with volumes from the gateway over iSCSI.

Storage gateway is still also communicating to the storage gateway endpoint which is running in AWS and it's still a public endpoint so this communication channel still runs over either the public internet or a public vif using a direct connect.

Where the difference starts is that the main location for the data is no longer on-premises instead the primary location for the data is in AWS specifically S3 so instead of the storage gateway having a local storage volume instead it has local cache and the primary data for all of these volumes that are presented by the storage gateway is now in S3.

This is critical to understand because it's the main difference between storage gateway running in volume stored mode and volume cached mode.

In volume stored mode all of the primary data is stored locally on the gateway.

In volume cached mode all of the data is stored in S3 and it's cached locally.

Now importantly when I'm talking about the primary data being stored in S3 it's actually an AWS managed area of S3.

This is only visible from the storage gateway console so you can't just open a bucket and look at the data because it's in this AWS managed part of S3 and if you think about it that wouldn't make sense because the data being stored is raw block storage it's not files or objects it's raw data from a volume.

Now you can still use it to create EBS snapshots and these snapshots will be copies of all the data that's on the volumes as your on-premises servers see them so in that sense it's the same as volume stored mode.

The key difference between volume stored and volume cached is the location of the data.

In volume stored mode the entire data is managed by the storage gateway and runs with it on-premises.

In volume cached mode the primary data runs from S3 and the only thing that's stored locally is the frequently accessed data.

Now this presents some really awesome architectural benefits because the only data being stored locally is cached data it means that we could potentially have hundreds of terabytes of data managed by the storage gateway and presented to servers and only consume a fraction of that locally in the form of cache storage and this allows for an architecture known as data center extension.

Imagine if the animals for live premises in this example was at capacity it only has a small server room maybe with one or two racks now imagine that the business knows long-term it needs to migrate all of its data through to AWS but right now it has urgent capacity issues.

Well what it can do is extend into AWS.

AWS becomes an extension of the animals for life business premises.

Storage in AWS appears to be on-premises but it's actually inside AWS.

Now up until now you might have thought that volume stored and volume cached are similar and they are they're very similar both work with volume so raw block storage both of them provide backups in AWS both are allow you to create EBS snapshots and volumes from these snapshots volume stored only really provides backup disaster recovery and migration capabilities it doesn't allow for this capacity or data center extension volume stored means data is accessible locally at local area network speeds with volume cached data is stored in AWS as the primary location any data written on premises is copied to AWS any data frequently accessed on premises is cached locally in the cache storage and so it too is accessible with local area network speeds anything which is not frequently accessed locally doesn't get cached locally and this means that it can be slower to access but it means capacity can be extended into AWS if you only have a hundred GB of storage for the storage gateway you can use it to provide access to hundreds of TB's of AWS stored data in gateway cached mode a single gateway can handle 32 TB per volume 32 volumes max for a total of one PB of storage so the common aspect of these two modes is that both work with volumes which are raw block storage where the difference is is that volume stored mode stores everything on premises and essentially uses AWS just for backups running in volume cached mode the storage gateway only caches a local copy of any frequently accessed data everything else is stored in AWS and this allows for a data center extension architecture which is what's on screen now for the exam if you see the keyword volume in a storage gateway question then it's going to be volume mode the difference between the two volume gateway types is whether you're going to be using it for either backups DR and migration or extension and with the information in this lesson it should now be clear which type of volume gateway you would pick based on those two different scenarios with that being said that's all of the theory that I wanted to cover in this lesson in the next lesson I'll be covering another mode of storage gateway which is tape mode also known as VTL mode so go ahead complete this lesson and then when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about the AWS Transit Gateway known as TGW.

Now this is a product that I remember being super excited about because at the time there was a massive hole in the hybrid network capability on the AWS platform.

Transit Gateway answered a lot of the complexity issues which plagued AWS in this space.

Now understanding why the Transit Gateway is such a valuable product means focusing both on the features it provides as well as how it can help evolve network architectures and that's something I'll attempt to do in this lesson.

Now we do have a lot to cover so let's jump in and get started.

The Transit Gateway is a network transit hub which connects VPCs to each other and to on-premises networks using site-to-site VPNs and Direct Connect.

It's designed to reduce the complexity of network architectures within AWS.

It's a network gateway object so another one that you need to remember and like other network gateway objects it's highly available and scalable.

The architecture is that you create attachments which is how Transit Gateway connects to other network objects within AWS and it's how it connects to on-premises networks.

Currently valid attachments include VPC attachments, site-to-site VPN attachments and Direct Connect gateway attachments.

Now to understand why Transit Gateway is required it's useful to compare a moderately complex network architecture and see how Transit Gateway affects that complexity.

So let's do that.

Let's use Animals for Life as an example and let's assume that the Animals for Life network has evolved and they have four VPCs.

So A, B, C and D as well as a corporate office.

Now we want to connect these together so that every point has connectivity to every other point in a fully highly available way.

Now we can use VPC peering connections to connect the four VPCs and these are already highly available and scalable.

But as you know they don't support transitive routing and so we need to create a full mesh between all VPCs.

So rather than four peering connections we'd need to have six.

That's already six connections that we need to plan, implement, manage and support but that's not everything that we need for this architecture.

We need the corporate office to be connected to all of the VPCs and because VPN routing isn't transitive we need a full mesh here as well.

And this means a customer gateway on the customer side and VPN connections between the VPCs and that customer gateway.

So that's a total of eight tunnels, two tunnels from each of the VPCs to the customer gateway of the Animals for Life corporate office.

So this ensures high availability at the AWS side.

But remember we need this architecture to be fully, highly available and we currently have a single point of failure, the customer gateway on the customer side.

While we have multiple availability zones at the AWS side all connecting back to the customer premises they all do so via this single customer gateway.

And so to make this architecture fully, highly available we need to add another customer gateway on the customer premises.

Ideally using a separate internet connection and ideally in a separate premises entirely and then have that customer gateway connected back to all four of these.

VPCs.

So that adds another eight tunnel connections.

Now I'm hoping at this point that you start to see the problem.

A full mesh network is complex.

It has a lot of admin overhead to implement and to maintain.

And what's more, it scales really badly.

The more networks are involved, the more networks get added each time a new network is added.

It's a problem which gets worse the more you scale.

So in addition to getting more complex that increase in level of complexity itself increases the more that you scale.

So this is not a solution that we can use much beyond this point.

If we add additional VPCs or additional customer premises, it gets really complex really quickly.

And that's where the transit gateway becomes really useful.

So let's have a look at how that changes this architecture.

Now using transit gateway, we start with the same basic architecture.

So for VPCs A, B, C and D and then the corporate premises with the two customer gateway routers.

This time though, we have a transit gateway which we create within the Animals for Life AWS account.

A transit gateway can use a site to site VPN attachment meaning it becomes the AWS side termination point for the VPN.

So rather than having to have connections between the corporate office and every VPC terminated on a virtual private gateway, instead each customer gateway only has to be connected back to this single transit gateway.

So we have the same level of high availability.

We still have the four tunnels connected from different availability zones at the AWS side to different customer gateways at the on-premises side.

So we don't lose any high availability with this solution, but we do reduce the number of VPN tunnels that are required.

With the transit gateway, you can also create attachments to VPCs and just like VPC interface endpoints that you used earlier in the course, you need to specify a subnet in each availability zone inside the VPCs that you want to use the transit gateway with.

And when you do that, it acts as a highly available inter-VPC router.

So one single transit gateway can route traffic between lots of different VPCs and this is a transitive routing capable device.

So we only need attachments from the transit gateway to VPC A, B, C and D and then all of the VPCs can talk to each other through the transit gateway.

And in addition to allowing full connectivity between all of the VPCs, because we have the VPN attachment, it means that all of the VPCs can communicate with the on-premises environment as well as the on-premises environment being able to communicate with all of the individual VPCs using this single network routing device, the transit gateway.

And in addition, you can also peer transit gateways with other transit gateways in other accounts in other regions.

So you can use this to peer with networks that themselves are connected to the peer transit gateway and those can be in other AWS regions and even across accounts.

And this is a really useful feature to create a global network within AWS.

Any cross-region data uses the AWS global network and so benefits from the more predictable latency rather than using the public internet.

And in addition, you can also attach a transit gateway to direct connect gateways if your business uses direct connect.

And so this allows you to use the transit gateway as well with physical, private networking connections into your business premises.

And I'll be covering this specific feature in a dedicated lesson elsewhere in the course.

Transit gateways come with a default route table, which is how traffic is routed between attachments.

But you can create a complex routing topology by using multiple route tables.

So before we finish this theory lesson, just some important considerations that you should be aware of for the transit gateway.

It does support transitive routing, which means that you don't need to create this full mesh topology.

You can have a single transit gateway with multiple attachments and it will orchestrate the routing of traffic between any of those attachments as long as appropriate routing is in place.

So you need route tables with routes on them in order for the transit gateway to route traffic between its different attachments.

But assuming you do, then it's capable of transitive routing.

Transit gateway can be used to create global networks.

I've just talked about that.

You can peer different transit gateways.

And again, you need to be aware of this for the exam.

You can share transit gateways between different AWS accounts using AWS RAM or resource access manager.

I've not covered that product yet in the course, but it's an AWS service which allows you to share products and services or components of products and services between different AWS accounts.

Again, you can peer transit gateways with different regions in the same or cross accounts.

So remember that one for the exam.

It doesn't have any limitations in terms of region or accounts.

You really can use it to create really complex, highly efficient routing topologies.

Now, overall, the thing to remember about the transit gateway is it offers much less complexity in terms of network architectures than without transit gateway.

So instead of having to use multiple VPC peers and then a full mesh topology in terms of VPN connections, you can use this one single resilient scalable, highly available device to perform transitive routing between all of your different networks.

And that results in a massive reduction of network complexity.

Now, there's going to be a demo lesson in this section of the course.

Well, you'll get the opportunity to implement a transit gateway inside your AWS account.

You'll get to experience using a transit gateway instead of VPC peering connections to link different VPCs together.

But with that being said, that's all of the theory that I wanted to cover in this lesson.

So go ahead, complete this video.

And when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about how you can use Direct Connect, PublicViffs and IPsec VPNs to provide end-to-end encrypted access to private VPC networks across Direct Connect.

Let's just jump in and get started covering this really useful feature of Direct Connect and virtual private gateways.

Now using a VPN gives you an encrypted and authenticated tunnel and this is true whether you use the public internet or run a VPN over Direct Connect.

By running a VPN over Direct Connect though you get that plus low latency and consistent latency.

You get great performance together with great security.

Now architecturally this uses a public vif and many students get confused by this because it provides access to a private VPC so why not use a private vif?

Well remember I said that a private vif gives access to private IPs only.

A public vif gives access to public zones meaning public IP addresses owned by AWS.

Well with a VPN what you're connecting to are public IPs which belong to a virtual private gateway or transit gateway and so to access these public IPs you need a public vif.

When you're thinking about vifs focus on what it is that you're trying to access and that should inform you whether you should use a public vif or a private vif.

Now a VPN is great because it's transit agnostic.

You can connect using a VPN to a virtual private gateway or a transit gateway over the public internet or over a public vif.

The VPN configuration is the same it's just the transit which differs public internet or public vif.

A VPN is end-to-end encryption from a customer gateway through to a transit or virtual private gateway.

Compare this to MaxEc which I've covered in another lesson of the course which is single hop based between the AWS DX router and whatever your cross connect is terminated into.

I say this so that you understand that a VPN and MaxEc are not competitors they do different things.

A VPN provides an end-to-end encrypted tunnel between a customer gateway and an AWS virtual private gateway or transit gateway and a MaxEc connection is between two hops in the same layer two local area network.

VPNs have wide vendor support getting a router which supports IPsec VPN is easier than getting a switch which supports MaxEc although this will change over time.

VPNs also have more cryptographic overhead versus MaxEc meaning VPN speeds tend to be very limited based on the hardware that you use.

MaxEc is much faster and designed for terabit or above network speeds.

Now one very common pattern for using an IPsec VPN and direct connect together is that a VPN can be provided immediately in minutes using software whereas direct connect takes time.

This means you can start with a VPN get your traffic flows working over that and then add a direct connect later either leaving the VPN in place to encrypt primary traffic over the direct connect or using it as a backup to the direct connect or both.

A common form of resilience in many of my clients is to have a normal internet connection over this you also run an IPsec VPN into AWS then you have a direct connect also running another IPsec VPN tunnel this way traffic is always encrypted.

The direct connect is the primary so the organization benefits from great performance and you have a backup in the form of a completely separate network connection and a completely separate IPsec tunnel.

Now visually the architecture of using a VPN and direct connect looks like this so this is a typical architecture we have two AWS regions on the left us east one at the top and AP southeast two in the middle we also have a direct connect location in AP southeast two in the middle of your screen and a business premises on the right when using a VPN with AWS in this case let's assume that we're going to use a virtual private gateway what actually happens is that two VPN endpoints are created within the AWS public zone in that region so one in each of two availability zones and these endpoints have public addressing this means that you can either connect from the customer site to these endpoints using the public internet for transit which means lots of hops as well as a fairly varied latency or you can create the IPsec VPN across the public VIF which means you'll benefit from direct connect low and consistent latency in both cases it's the same encrypted tunnel between the customer on premises gateway and the AWS VPN endpoint only using a direct connect means using a public VIF as transit so you'll benefit from the performance improvements that direct connect provides you can even connect to VPN gateways in other AWS regions using the same public VIF across the AWS global network and this is often a great way to provide global encrypted transit between VPCs and your business network what I want you to take away from this lesson is that IPsec running over a direct connect it doesn't compete with max sec VPN is when you need end-to-end encryption of data from AWS to on-premises networks using something which can work equally as well over the public internet and the public VIF it also means that you can connect to VPCs in remote regions using the same grade of equipment focus on understanding why we're using a public VIF with this architecture because we're connecting from the customer gateway to a public IP or public IPs which are provided by the virtual private gateway or transit gateway because we're connecting to public IPs we need to use a public VIF with that being said that's everything I wanted to cover about this topic go ahead and complete this video and when you're ready I look forward to you joining me in the next.

Welcome back.

As an architect, you need to understand the impacts of failures at various points within the DX architecture and the way that it integrates with your on-premises networks.

So let's jump in and get started because we've got a lot to cover.

Now before we start looking at how resilience works with Direct Connect, let's review an architecture which is not resilient.

Now, worryingly, this is actually the way I've seen most people provision Direct Connect.

A typical DX deployment has a number of physical components.

First, the AWS region.

Think of this as the actual infrastructure that AWS uses to deliver services in that region.

Secondly, a Direct Connect or DX location, which is typically a data center in the geographic area that the region is in.

Now there are often multiple DX locations in a single region and these are generally located within a major data center in a metro area.

And the last typical component in most architectures is the customer premises.

So your office buildings or self-managed data centers.

Now architecturally, I want you to think of the AWS region as a separate thing in terms of Direct Connect architecture.

So an AWS region is connected to all of the DX locations in that region using multiple high-speed network connections.

Now you can assume that this part is always highly available.

So while the region and DX location are conceptually different, they're always connected with high performance, highly available networking.

Now inside the DX location, which remember is a data center, are a collection of AWS DX routers and these conceptually are connected to the AWS region.

So picture a DX router, which is the exit point of the AWS network.

When you order a DX connection within your AWS account, what you actually receive is a port on a DX router at a DX location.

Now ideally, you'll also have equipment at this DX location and this is referred to as the customer DX router.

If not, you'll need to purchase a service from a communications provider and use one of their routers, which is also going to be at the DX location.

But in either case, there's another router inside the DX location, the customer or provider DX router.

So when you order a DX connection, you're allocated a port on the AWS DX router and you need to arrange a connection between this port and a port on your customer DX router or provider DX router.

And this process, this cable is called a cross connect and it's a single cable between both of these routers.

In addition to this, generally you'll also want to connect the DX location back to your company network.

If you're a large company, you might have lots of capacity at the same data center that the DX location uses.

But if not, you'll need to extend this back into your own on-premises network.

Generally, you'll have a customer premises router and you'll pay a carrier or a telco provider to extend the direct connect from the customer or provider DX router all the way back through to your on-premises network.

So to summarize, the AWS region is linked in a highly available way to one or more DX locations.

The DX location houses the AWS DX routers in addition to your or a provider's DX router and you cross connect from the AWS DX router into your DX router with a physical cable.

And then all of this is extended to your on-premises environment.

So what can go wrong with this architecture?

Well, there are actually seven single points of failure on this diagram.

There are actually more, but seven that I would be directly concerned about if I was implementing direct connect.

The first is the entire DX location could fail.

You can have power failures, buildings can and often do collapse.

The AWS DX router could fail either in isolation or along with the building.

The same for the cross connect.

It's just a cable.

Cables do fail.

Your DX router could fail and what if you don't have an on-site spare?

The extension from the DX location to your on-premises environment.

It's just a cable.

It probably goes under the road or across above ground cables and any engineering works can potentially be a risk to the stability of that cable.

You might also have a failure of your actual customer premises environment or your customer premises router within that environment could also have a hardware failure.

When people think about direct connect for some reason, they have a perception that it's a resilient product.

It's actually not resilient in any way by default.

It's a product which is based on lots of physical components which each depend on each other.

But it's also a product that's designed to be flexible.

So it can actually be made into a super resilient service.

Let's look at that and now that you know about the architecture details, I can speed up and zoom out a little.

Now we can improve the resilience of the architecture by provisioning multiple DX ports.

So we start with the same architecture at a high level.

The AWS region on the left, the DX location in the middle and the customer premises on the right.

Now there are actually multiple points of connection within each DX location.

If you have multiple routers in the DX location, you can configure multiple cross-connects into multiple DX ports.

And from there you can extend those into multiple customer premises routers using extensions.

So this architecture has two AWS DX routers, two customer DX routers and two customer premises routers.

So when you're ordering direct connect, if you order two direct connects into the same DX location, then AWS will provision these onto separate DX routers and that gives you a level of resilience.

Now this architecture adds significant benefits because we have the two independent DX ports, two DX routers and two customer routers.

The architecture can tolerate a failure of a router in either of those two paths or a failure of one of the extensions and still continue operating.

The design does have some problems though.

Some single points of failure do still exist.

If the DX location itself fails, then connectivity is lost and since all connectivity goes via a single customer location, if that single customer location fails, then connectivity to the AWS platform is also lost.

So we still have two fairly major single points of failure, the two locations that are involved in this private networking.

We also have a potentially hidden single point of failure with this architecture and this occurs because the extensions between the DX location and the single customer premises location could in theory travel via the same physical cable route.

So if you order multiple connections between two physical locations, then generally the cable route for both of those connections could be the same.

This isn't always the case but it's something to be very aware of.

I've seen many businesses build this type of architecture.

They assume a high level of resilience only to find that both of their independent cables enter their building under the same sidewalk and both of those cables were broken by roadworks occurring on that same sidewalk.

So to prevent this, we need another step of evolution in terms of resilience.

We can't have any single points of failure if the private networking is important to our organization.

So let's look at a better version of this architecture, another evolution in terms of resilience.

So at a high level, we still have the AWS region on the left but now we have two independent customer premises on the right.

So two completely different buildings, ideally two buildings that are spread out geographically.

In addition, we have two different DX locations.

In each DX location, we're going to provision one DX port that's cross-connected into one customer DX router and each of these is going to be extended into a different customer router in a different customer premises.

If we architect the solution this way, it offers much better resilience than the previous architecture.

We've got two different DX locations, meaning the architecture can tolerate the failure of one of these locations and still continue to operate.

Because there are two customer premises and two customer routers, it can also tolerate the failure of hardware and location at the customer side.

The only risk of outage is if an entire location fails and then the hardware in the remaining path also fails.

So with this architecture, if we had hardware failure in DX location one, the solution would continue to run.

If the entire DX location one failed, the solution would continue to run.

If both DX location one and the customer premises one failed, the solution would still provide connectivity.

Only if we had all of that fail and then in addition, if we had hardware failure in DX location two or customer premises two, would the connectivity be interrupted.

But we can take this one step further and implement an architecture designed for extreme levels of resilience.

This next design offers maximum resilience.

We still have the AWS region on the left, the two DX locations in the middle and the two customer premises environments on the right.

However, this time in each of the DX locations, we have two DX ports on separate equipment.

And then in each location, we also have two customer DX routers.

This provides a level of resilience within each location covering hardware failure and resilience against the failure of an entire location.

So at each location, we have a pair of cross-connects between the AWS DX router and the customer DX router.

And then these are all extended to dual customer routers at each of the customer premises locations.

So this architecture gives us extreme levels of resilience.

We have high availability from a direct connect location perspective.

One can fail and the connectivity is still active because we have infrastructure in both.

Inside each DX location, we have a pair of both AWS DX routers and customer DX routers.

So even if one DX location fails, we could still lose one pair of those in the remaining DX location and still the connectivity would be fine.

The extensions from the DX location to the customer premises because they're going between two completely different locations at both sides will generally follow two completely separate routes.

So this is because the starting and the end points of those connections are to different physical buildings, which means the extensions themselves and the customer premises are going to be highly available.

And inside each of the customer premises because we're using multiple customer routers, these are also highly available.

Now this is relatively complex network architecture and the thing that I want you to take away from this lesson from an exam perspective and if you intend to use direct connect in real world projects is that direct connect is a physical technology and so it is not resilient in any way unless you architect it that way.

So if you just provision a single direct connect, it will be a single port on a single DX router at a single DX location.

You'll cross connect it to a single customer DX router, extend it to a single customer premises with a single customer router.

Each step of that process will be a single pointer failure.

If you want something that is highly available and you need to use direct connect, then you need to consider one of the more resilient solutions that I've demonstrated in this lesson.

Now you can also use site to site VPN as a backup for direct connect and I've explained exactly how this works in another lesson in this section of the course, but generally if you do need to use direct connects only for a design then you should definitely review all of these highly available architectures rather than provisioning a simple single direct connect.

Now with that being said that's all of the theory and the architecture that I wanted to talk about in this lesson so go ahead complete the lesson and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about AWS Direct Connect.

A direct connect is a physical connection into an AWS region.

If you order this via AWS and this connection is either a 1 gig, 10 gig or 100 gig at the time of creating this lesson.

There are other ways to provision slower speeds but I'll be covering those in a dedicated lesson later in this section of the course.

The connection is between a business premises, a direct connect or DX location and finally an AWS region.

And I'll show this architecture visually on the next screen.

Conceptually think of three different physical locations.

So your business premises where you have a customer premises router, a DX location where you also have other equipment such as a DX router and maybe some servers and then finally an AWS region such as US East 1.

When you order a DX connection what you're actually ordering is a network port at the DX location.

AWS provide a port allocation.

That's it.

They also authorize you to connect to that port and I'll detail that process very soon.

But a direct connect which is ordered directly from AWS doesn't actually provide a connection of any kind.

It's just a physical port.

It's up to you to connect to this directly or arrange the connection to be extended via a third party comms provider.

Now the port has two costs.

You have an hourly cost based on the DX location together with the speed of the port and then there's a charge for outbound data transfer.

Inbound data transfer is free of charge.

Now there are a couple of important things to keep in mind about direct connect.

The first is the provisioning time.

AWS will take time to allocate a port and then once allocated you will need to arrange connection into that port at the DX location.

If you want to connect the DX location to your business network and haven't already done so then you might be looking at weeks or months of extra time for the physical laying of cables between the DX location and your business premises.

So keep that in mind.

Now because it's a physical cable there's no built-in resilience.

If the cable is cut the cable is cut.

You can design in resilience by using multiple direct connects but that's something that you have to layer on top.

Direct connects provide low latency and that's because data isn't transiting across the public internet like with say a VPN.

It also provides consistent latency as you're using a single physical cable at best or a small number of private networking links at worst.

In any case if you need low and consistent latency for an application then direct connect is the way to go.

In addition it's also the best way to achieve the highest speeds for hybrid networking within AWS.

As above it can be provisioned with one, ten or a hundred gigabit speeds and because it's a dedicated port you're very likely to achieve the maximum possible speed of that port.

Now contrast that with say an IPsec VPN which uses encryption so requires processing overhead and transits over the public internet and you can see how direct connect is going to give higher more consistent speeds.

And then lastly direct connect can be used to access AWS private services running in a VPC and AWS public services.

It cannot be used to access the public internet unless you add a proxy or other networking appliance to do that on your behalf.

So visually this is the architecture of direct connect.

We start on the right with your business premises and inside it will be some kind of customer premises router or firewall.

This might be the same router which is connected to your internet connection or it might be a new dedicated DX capable router.

And I'll talk more about what this means in an upcoming lesson.

Additionally you're going to have some staff in this case, Bob and Julie.

Now next in the middle we have a DX location.

This part is often confusing because this is not a location which is actually owned by AWS.

It's not an AWS building at all.

It's often a regional large data center within which AWS rent space but your business might also rent space together with other businesses.

Inside this DX location conceptually there's going to be an AWS cage which is an area owned by AWS.

And this contains one or more DX routers known as AWS DX routers which are the endpoints of the direct connect service.

Often you'll also rent space at this DX location and this is known as the customer cage.

And this is where things can start to differ.

If you're a large organization you might rent this space directly in which case it might have some of your other infrastructure together with a router.

So this is known as the customer DX router.

Now if you're a smaller organization then this cage might belong to a communications partner and this is known as the comms or communications partner cage.

So if you don't have space within a DX location the communications partner does and they extend any connections from this DX location into your business premises.

The key thing to understand about direct connect is that it's a port allocation.

You order a direct connect from AWS to specific DX location and you're allocated a DX port.

This needs to be connected physically with a fiber optic cable to another port in the DX location.

So this is either your router in your cage at the DX location or it's a communication partners router which is also located at the DX location.

Now in either case you're going to have a corresponding port within the DX location either your equipment or a comms provider equipment.

And between these two ports you're going to order a cross connect.

It's a connection between the direct connect port within the AWS cage in the DX location and either a port on your router or a communications partner router also within the DX location.

And this is really important to understand whether you have equipment within the DX location directly or whether you purchase this capacity from a communication provider.

From a communications partner you will be given a port within the DX location and it's to this port that you will use a cross connect.

And this is a physical cable that connects the AWS DX port with your port on either your or a communication partner router.

Now if you are using a communications partner then this link can be further extended through to your customer premises.

But in either case you do have to have a port within either a customer cage or a comms partner cage at the DX location because this is what you will use to connect using a cross connect through to the AWS DX port.

Now at the left side we have an AWS region in this case AP Southeast 2.

And then inside this a VPC with a private subnet containing some services.

And then we have the AWS public zone and also some example services in this case SQS and elastic IP addresses for that region and S3.

Now the AWS region is AWS owned infrastructure.

It might be running from the same physical facilities as the DX location or it might be different.

But in either case it's connected with multiple resilient high speed network connections.

Conceptually just think about it as something which is always connected with one or more local DX locations.

So that's the physical architecture and I'm going to be going into much more detail in upcoming lessons elsewhere in the course.

Logically we have to configure virtual interfaces over this single physical connection and these are called VIFs and I'll be covering these in depth very soon.

There are three types of VIFs.

First transit VIFs which have a very specific use case and I'll be talking about these in detail elsewhere in the course.

Second we have public VIFs which are used to access the AWS public space services and a public VIF being a virtual interface runs over the end to end direct connect from your customer outer through to the customer DX router into the AWS DX router and then into the public part of the AWS region.

You also have private VIFs which run over the direct connect but connect into virtual private gateways which are attached to a VPC and these provide access to private AWS services.

At this point though that's everything I wanted to cover so go ahead and complete this lesson and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I'm going to be stepping through the architecture of AWS site-to-site VPN.

For the exam and if you're designing solutions for real-world production usage, understanding VPNs and how they can be used within AWS is essential.

They offer the quickest way to create a network link between an AWS environment and something that's not AWS.

And this might be on-premises, another cloud environment or a data centre.

Now we've got a lot to cover so let's jump in and get started.

A site-to-site VPN is a logical connection between a VPC or virtual private cloud and an on-premises network.

And this connection is encrypted in transit using IPsec, which is important because it runs over the public internet in most cases.

Now there's a common exception to this when you run a VPN over the top of a direct connect and we'll be covering this later in this section.

But assume, unless written otherwise, that a VPN is running over the public internet.

Now a site-to-site VPN can be fully, highly available, assuming that you design and implement it correctly.

This is really important to understand for the exam and real-world usage.

So I'm going to be covering it as a priority in this lesson.

There are a few points of failure within the VPN architecture and you need to understand them all.

Site-to-site VPNs are also quick to provision.

Assuming you understand all the steps and have all the skills, you can have a site-to-site VPN up and running in less than an hour.

And try and remember this because it's in contrast to the long provisioning times for physical connections like direct connect, which we'll talk about later in this section.

Now there are a few components involved in creating a VPN connection that you need to be aware of.

First, and this goes without saying, the VPC.

VPNs connect VPCs and private on-premise networks.

So it's logical that the VPC is one important building block of the wider connection architecture.

Second, we've got the virtual private gateway or VGW and this is another type of logical gateway object, which can be the target on route tables.

It's something that you create and associate with a single VPC and it's the target on one or more route tables.

Next, we've got the customer gateway or CGW and this can actually refer to two different things.

It's often used to refer to both the logical piece of configuration within AWS and the thing that that configuration represents, a physical on-premises router which the VPN connects to.

So when you see CGW mentioned, it's either the logical configuration in AWS or it's the physical device that this logical configuration represents.

And then the last component is the VPN connection itself which stores the configuration and it's linked to one virtual private gateway and one customer gateway.

So this is how we create the network virtual connection between these two locations.

Now, later in this section, I'm going to be showing you how to create a VPN connection, but for now, I want to focus on the architecture and the theory.

So let's look visually at how VPNs are architected.

First, I want to cover a simple implementation of a site to site VPN so that you're comfortable with the architecture.

So on the left, we have the Animals for Life VPC.

It's a simplified version with three private subnets in availability zone A, B and C.

Next, we've got the AWS Public Zone where AWS public services operate from.

The public internet directly connected to that and then finally on the right, the Animals for Life corporate office using an IP range of 192.168.10.0/24.

Now, step one for creating a VPN connection is to gather all of the required information.

We need the IP address range of the VPC that will be connecting to the on-premises network and we'll also need the IP range of the on-premises network itself and the IP address of the physical router on the customer premises.

Once we have all of this information, then we can create a virtual private gateway and attach it to the Animals for Life VPC.

The virtual private gateway is a logical gateway object within AWS and so it can be the target of routes just like any other gateway object.

Now, within our on-premises environment, we're going to have a customer premises router and this will have an external IP address.

And for this router, we create a customer gateway object within AWS.

This is a logical configuration entity that represents this physical device on our customer premises.

In this case, we need to define its public IP address so that the logical CGW entity matches the physical router.

Behind the scenes, the virtual private gateway is actually a highly available gateway object.

Just the same as earlier in the course when you configured internet gateways.

All you had to do was create the gateway and associate it with a VPC and a virtual private gateway is just the same.

Behind the scenes, a virtual private gateway actually has physical endpoints.

These are devices in different availability zones each with public IP version 4 addresses.

And this means that the virtual private gateway is fully, highly available by design.

An availability zone can fail and if it affects one of the physical endpoints, the other will still function.

But that doesn't mean that the whole thing is highly available and I'll detail that during the remainder of this lesson.

The next step is that we need to create a VPN connection inside AWS and there are different types of VPN connection.

There are static and dynamic VPNs.

For now, we're going to create a static one and I'll be explaining the differences between the two later in this lesson.

Now, when creating a VPN connection, you need to link it to a virtual private gateway and this means that it can use the endpoints which that virtual private gateway provides.

You also need to specify a customer gateway to use and when you do, two VPN tunnels are created.

One between each endpoint and the physical on-premises router.

A VPN tunnel is an encrypted channel through which data can flow between the VPC and on-premises network or vice versa.

As long as at least one of these VPN tunnels are active, then the two networks are connected.

So in this particular case, we've got one VPN connection that's using the two endpoints of the virtual private gateway and both of those are connected back to our one customer gateway.

So we have a partially highly available design.

If one of the AZs on the AWS side fails, then the other endpoint will continue functioning.

So at least one of these tunnels will be active.

Now, because this is a static VPN, it means that we have to statically configure the VPN connection with IP addressing information.

So we have to tell the AWS side about the network range that's in use within the on-premises network and we have to configure the on-premises side so that it knows the IP address range that the AWS side uses.

And this means that traffic can flow from the VPC via the VPC router through the virtual private gateway over the tunnels to the on-premises network and back again.

Now, as I just mentioned, this design from an overall perspective is actually not fully highly available, but there is still one single point of failure.

And that single point of failure is the customer on-premises router.

If this fails, then the whole VPN connection fails.

Even though at the AWS side, it is highly available, all of the connections currently terminate into the single customer on-premises router.

At the AWS side, there are two tunnels to separate hardware in separate availability zones, but this doesn't matter if the customer side fails because all of the tunnels terminate into the same single point of failure.

And this is known as partial high availability.

It's highly available on the AWS side, but suffers from a single point of failure on the customer side.

So it's not a fully highly available solution.

It is actually pretty simple to resolve this to modify this design so that it is fully highly available.

And let's look at that next.

Moving to a fully highly available solution means adding another on-premises customer router using a second internet connection and ideally doing all of this in a separate building.

Once you have this second resilient connectivity method, then you can create an additional VPN connection at the AWS side.

Now, behind the scenes, this actually creates two more physical endpoints which are managed by the virtual private gateway.

And each of those has their own public IP addressing.

So this new VPN connection, it would be linked to the same virtual private gateway at the AWS side, but to the new customer gateway.

So it would establish another pair of VPN tunnels between the two new endpoints and that additional customer gateway.

Now, this means architecturally, we're in the situation where the virtual private gateway is now highly available.

So it's highly available as a logical entity.

It's using endpoints which are located in different availability zones and so it can withstand physical failure.

But now in addition, at the customer side, we're now using multiple pieces of hardware with multiple internet connections, ideally in separate buildings.

And that means this is a fully highly available solution.

It's got two VPN tunnels connecting each customer premises router to two VPC endpoints in separate availability zones.

And then that configuration is repeated again with a second customer gateway.

So either of the customer gateways can fail, either of the availability zones can fail.

And still the VPC and on-premises network will have connectivity.

Now, before we finish, I just want to talk about the differences between static and dynamic VPNs.

Now, a dynamic VPN uses a protocol called BGP known as the border gateway protocol.

And that's important because if your customer router doesn't support BGP, then you can't use dynamic VPNs.

So conceptually, this is a traditional VPN architecture.

A VPC subnet on the left, a VPC router middle left, a virtual private gateway middle right, and then an on-premises environment with a customer router on the right.

This architecture is the same based on both types of VPNs, so static VPNs and dynamic VPNs.

At a high level, the difference is how routes are communicated.

So a static VPN uses static networking configuration.

Static routes are added to the route tables and static networks have to be identified on the VPN connection.

The benefit of using a static VPN is that it's simple.

It just uses IPsec and because of that, it works almost anywhere with any combination of routers.

You are really restricted on things like load balancing and multi-connection failover.

So if you need any advanced high availability, if you need to use multiple connections, if the VPNs need to work with Direct Connect, which we'll talk about later in this section, then you really do need to be using dynamic VPNs.

With dynamic VPNs, as I just mentioned, you're using a protocol called BGP or the Border Gateway Protocol.

Now this is a protocol which lets routers exchange networking information.

And so if you have a dynamic VPN, you're creating a relationship between the virtual private gateway and the customer router.

Over this relationship, they can both exchange information on which networks are at the AWS side and which are at the customer side.

In addition, they can communicate the state of links and adjust routing on the fly.

And that allows for multiple links to be used at once between the same locations.

So this is why dynamic VPNs are able to use really high end, highly available VPN architectures because they can communicate the state of the links between the virtual private gateway and the customer gateway.

Now with dynamic VPNs, routes can still be added to the route table statically.

Or you can make the entire solution fully dynamic by enabling a feature called route propagation on the route tables in the VPC.

And when you enable route propagation, it means that while any VPNs are active, any networks that these VPNs become aware of, so the on-premises networks in this example are automatically added as dynamically learned routes on the route tables.

So instead of having to statically enter routes on each and every route table, if you enable routes propagation, they will learn these routes from the virtual private gateway whenever any VPNs are active.

So any virtual private gateways which learn any routes using BGP, they can automatically and dynamically be added onto route tables on a per route table basis if you enable route propagation.

Now whichever method you decide on, there are a number of key considerations that you need to be aware of.

First, there is a speed cap for VPNs.

A single VPN connection with two tunnels has a maximum throughput of 1.25 gigabits per second.

Now this is an AWS limit.

You would also need to check the speed supported by your customer router because VPNs use encryption.

There's a processing overhead on encrypting and decrypting data and at high speeds, this overhead can be significant.

Now the speed limit is something that you should remember for the exam because it's often something which makes selecting between VPNs and something else significantly easier.

So if you need more than 1.25 GB per second, then you can't use VPNs.

Now you also need to be aware that there is a cap for the virtual private gateway as a whole.

So for all VPN connections connecting to that virtual private gateway and that's also 1.25 GB per second.

Another consideration for VPNs is latency.

The VPN connection transits over the public internet and depending on the quality of your internet connection, there may be many hops between you and the AWS VPN endpoints.

Each hop adds latency and variability.

So if you care about these as a priority, maybe you're running an application which is really latency sensitive.

You might want to look at something else like Direct Connect which we'll be covering later in this section.

Again, latency is often a selection criteria to pick between VPNs and something else in the exam.

Now in terms of costs, VPNs have an hourly cost to operate.

There's a data transfer charge to transfer data out.

And because you're using the internet to transit data, your on-premises internet connection is also going to be used by the VPN.

So if you have any data caps on your internet connection, this is something to keep in mind, especially if you're going to be using a VPN to transfer lots of data.

Now one of the benefits of VPNs and this is important for the exam is that they are very quick to set up.

Sometimes taking hours or less because it's all software defined.

An IPsec is supported on pretty much all hardware at this point, even consumer grade routers.

It is worth keeping in mind though that to use dynamic VPNs you will need BGP support which is much less common.

But when comparing VPNs to anything else, VPNs are almost always quicker to set up versus other private connection technologies.

VPNs can also be used as a backup for a physical connection such as Direct Connect Rather than needing to provision two physical connections for true high availability, you can use one physical connection as the primary and use VPNs as the secondary.

VPNs can also be used with physical technologies though, for example Direct Connect.

So you can use a VPN at the start because they're quick to set up and provision.

And then you can lodge a request to provision a Direct Connect which is added later.

So a Direct Connect can take much longer to provision sometimes weeks or months, but you can use a VPN to set up that initial connectivity and then either replace it further down the line as the Direct Connect comes online or you can run both of them together for high availability.

Now VPNs can also be used over the top of Direct Connect to add a layer of encryption, but I'll be covering that in the Direct Connect lesson.

For now though that's all of the theory that I wanted to cover, so go ahead complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to cover the theory and architecture for the simple notification service or SNS.

SNS is a key component of many architectures within AWS so it's one which you need to fully understand.

So let's jump in and get started because I really want to make sure that you understand SNS end to end.

Simple notification service or SNS is a highly available, durable, secure, pub/sub messaging service.

It's a public AWS service meaning to access it you need network connectivity with the public AWS endpoints.

But the benefit of this is that it's accessible from anywhere that has that network connectivity.

What it does at a high level is coordinate the sending and delivery of messages and messages are payloads which are up to 256 kilobytes in size.

Now I'm not mentioning the size because you need to know it exactly for the exam but more so so that you understand that you can't send an entire cat movie using the service.

Architecturally, messages are not designed for large binary files.

Now the base entity of SNS is the SNS topic and it's on these topics where permissions are controlled as well as where most of the configuration for SNS is defined.

SNS has the concept of a publisher and a publisher is the architectural name for something which sends messages to a topic.

With a pub/sub architecture publishers send things into a topic.

Now as well as publishers, each topic can have subscribers and these by default receive all of the messages which are sent to the topic.

Now subscribers can come in many different forms.

We've got things like HTTP and HTTPS endpoints, email addresses which can receive the message, SQS queues where each message is added to the queue as it's sent to the topic.

Topics can also be configured with mobile push notification systems as subscribers so that messages sent to a topic are delivered to mobile phones as push notifications or as SMS messages.

Even Lambda functions can be subscribed to a topic so that that Lambda function is invoked as messages are sent into the topic.

SNS is used across AWS products and services.

CloudWatch uses it when alarms change state.

CloudFormation uses it when stacks change state.

Auto scaling groups can even be configured to send notifications to a topic when a scaling event occurs.

SNS is one of those subjects which is a lot easier to understand if you look at it visually.

So let's move on to an architecture diagram.

Now the SNS service as I just mentioned is a public space AWS service.

It operates from the AWS public zone.

And because of that the service can be accessed from the public internet assuming the entity trying to access it has the relevant AWS permissions.

And assuming that a VPC is configured to be able to access public AWS endpoints then SNS can be accessed from a VPC as well.

SNS as a service runs from this public zone and you can create topics inside of SNS.

For each topic a wide variety of producers so external APIs running on the public internet or CloudWatch or EC2 or auto scaling groups or CloudFormation stacks and many other AWS services can publish messages into a topic.

The topic has subscribers and things can be subscribers and producers at the same time for example APIs.

Any subscribers by default will receive all of the messages sent to the topic by publishers.

But it's possible to apply a filter onto a subscriber which means that subscriber will only receive messages which are relevant to its particular functionality.

Another interesting architecture that I want to comment on just briefly is the fan out architecture.

And this is when you have a single SNS topic with multiple SQSQs as subscribers and we'll be covering SQSQs later in this section.

But this is a way that you can create multiple related workloads.

So if for example a message is sent to an SNS topic when a processing job arrives that message can be added to multiple SQSQs which are configured as subscribers for that SNS topic.

Now each of these Qs might perform the same related processing but using the pet tube as an example might work on a different variant.

So there might be processing a different bitrate or a different video size.

So using fan out is a great way of sending a single message to an SNS topic representing a single processing workload and then fan that out to multiple SQSQs to process that workload in slightly different and isolated ways.

Now the functionality that's offered by SNS is pretty important.

It really is a foundational service for developing application architectures within the AWS platform.

So SNS offers delivery status.

So with a number of different types of subscribers you can confirm the status of delivery of messages to those subscribers.

An example of some subscriber types that do support delivery status is HTTP or HTTPS endpoints, Lambda and SQS.

As well as delivery status SNS also supports delivery retries so you've got the concept of reliable delivery within SNS.

SNS is also a highly available and scalable service within a region.

So SNS is a regionally resilient service.

So all the data that's sent to SNS is replicated inside a region.

It's scalable inside that region so it can cope with a range of workloads from nothing all the way up to highly transactional workloads and it's also highly available.

So if particular availability zones fail then an SNS topic will continue to function.

SNS is also capable of server-side encryption or SSE which means that any data that needs to be stored persistently on disk can be done so in an encrypted form.

So this is important if you do have any requirements which mandate the use of on disk encryption.

Now SNS topics are also capable of being used cross account just like S3 buckets.

You can apply a resource policy.

In the case of an SNS topic this is a topic policy.

It's exactly the same.

It's a resource policy that you apply to the resource, the SNS topic and you can configure from a resource perspective exactly what identities have access to that topic.

Now you'll need to know all of this architecture relating to SNS for the exam.

An SNS really is one of those topics that you'll be using extensively as you deploy projects inside AWS but for now that's everything that you need to know about SNS.

You will as we go through the course get more and more experience with the products but I wanted to illustrate all of the important pieces of architecture at this point.

So go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.