äžèŠãããšãããåé¡ã¯éåžžã«åçŽã«æããŸãã
AWSã®è²»çšãæ°ãæããšã«ç°¡åã«åé¡ããããã§çµäºã§ããŸãã EC2ã§1äžãã«ãS3ã§1000ãã«ããããã¯ãŒã¯ãã©ãã£ãã¯ã§500ãã«ãªã©ã ããããäœãéèŠãªæ¬ èœããããŸã-補åãšéçºã°ã«ãŒãã®çµã¿åããããã³ã¹ãã®å€§éšåãå ããŠããŸãã
ãããŠãæ°çŸã®ã€ã³ã¹ã¿ã³ã¹ãšæ°çŸäžã®ã³ã³ãããå€æŽãããå¯èœæ§ãããããšã«çæããŠãã ããã ããã«ãæåã¯åçŽãªåæåé¡ã®ããã«èŠãããã®ããæ³åã絶ããã»ã©è€éã«ãªããŸãã
ãã®èšäºã®ç¶ãã§ã¯ãç§ãã¡ã䜿çšããäžé£ã®ããŒã«ã«é¢ããæ å ±ãå ±æããããšæããŸãã ã€ã³ã¹ã¿ã³ã¹ãæ°åãããæ°äžåãããã«é¢ä¿ãªããAWSã³ã¹ãã®åææ¹æ³ã«é¢ããããã€ãã®ã¢ã€ãã¢ãæäŸã§ããããšãé¡ã£ãŠããŸãã
ã補åã©ã€ã³ãã«ããã°ã«ãŒãå
AWSã§å€§èŠæš¡ãªæäœãè¡ãå Žåããããã2ã€ã®åé¡ã«æ¢ã«çŽé¢ããŠããã§ãããã
第äžã«ãéçºã°ã«ãŒãã®1ã€ãçªç¶äºç®ãå¢ãããå Žåãæ°ä»ãããšã¯å°é£ã§ãã
AWSã®è«æ±æžã¯æé¡10äžãã«ãè¶ ããŠãããåAWSã³ã³ããŒãã³ãã®ã³ã¹ãã¯æ¥éã«å€åããŠããŸãã ç¹å®ã®é±ããšã«ã5ã€ã®æ°ãããµãŒãã¹ãå±éããDynamoDBã®ããã©ãŒãã³ã¹ãæé©åããæ°çŸã®æ°ããã¯ã©ã€ã¢ã³ããæ¥ç¶ã§ããŸãã ãã®ç¶æ³ã§ã¯ã1ã€ã®ããŒã ãå æãããEC2ã«ä»æ20,000ãã«å€ãè²»ããããšãã泚æãéããã¡ã§ãã
第äºã«ãæ°ãã顧客ãµãŒãã¹ã«ãããè²»çšãäºæž¬ããããšã¯å°é£ã§ãã
æ確ã«ããããã«ãåœç€Ÿã®ã»ã°ã¡ã³ãã¯ãããŒã¿ãŠã§ã¢ããŠã¹ãS3ããŸãã¯äŒæ¥ã®å éšæ å ±ã·ã¹ãã å ã®ãµãŒãããŒãã£ããŒã«ã«åæããŒã¿ãéä¿¡ããåäžã®APIãæäŸããŠããŸãã
顧客ã¯å¿ èŠãªãã©ãã£ãã¯éãšäœ¿çšãã補åãéåžžã«æ£ç¢ºã«äºæž¬ããŸããããã®äºæž¬ãç¹å®ã®éé¡ã«æç®ããéã«åžžã«åé¡ãçºçããŠããŸãã çæ³çã«ã¯ãã100äžã®æ°ããAPIåŒã³åºãã«ã¯Xãã«ããããããã¯ã©ã€ã¢ã³ãããå°ãªããšãYãã«ãååŸããå¿ èŠããããŸããã
ãã®åé¡ã®è§£æ±ºçã¯ãã€ã³ãã©ã¹ãã©ã¯ãã£ãã補åã©ã€ã³ããšåŒã¶ãã®ã«åå²ããããšã§ããã ç§ãã¡ã®å Žåããããã®æ瀺ã¯æ¬¡ã®ããã«æŒ ç¶ãšå®åŒåãããŠããŸãã
- çµ±åïŒã»ã°ã¡ã³ãããããŸããŸãªåæãããã€ããŒã«ããŒã¿ãéä¿¡ããã³ãŒãïŒã
- APIïŒã¯ã©ã€ã¢ã³ãã©ã€ãã©ãªããã»ã°ã¡ã³ãå ã®ããŒã¿ãåä¿¡ãããµãŒãã¹ïŒã
- ããŒã¿ãŠã§ã¢ããŠã¹ïŒã»ã°ã¡ã³ãããŒã¿ããŠãŒã¶ãŒããŒã¿ãŠã§ã¢ããŠã¹ã«ããŒããããã€ãã©ã€ã³ïŒã
- Webãµã€ããšCDNã
- å éšã·ã¹ãã ïŒäžèšã®ãã¹ãŠã®å ±éããžãã¯ããã³ãµããŒãã·ã¹ãã ïŒã
ãããžã§ã¯ãå šäœãåæãããšã ããã§ãã¹ãŠã枬å®ããããšã¯ã»ãšãã©äžå¯èœã§ãããšããçµè«ã«éããŸãã ã ãã®ä»£ããã«ãã¢ã«ãŠã³ãã®ã³ã¹ãã®äžéšãããšãã°80ïŒ ã远跡ãããããã®ã³ã¹ããæåããæåŸãŸã§è¿œè·¡ããã¿ã¹ã¯ãèšå®ããŸãã
äŒæ¥ãã¢ã«ãŠã³ãã®80ïŒ ãåæããããšã¯ã100ïŒ ãç®æããŠããŒã¿åéã®æ®µéã§è¡ãè©°ãŸã£ãŠçµæãåºããªãããšããã䟿å©ã§ãã ã³ã¹ãã®80ïŒ ãã«ããŒããããšïŒãããã§ååããšèšãæ欲ïŒã«ãããäœåºŠãäœåºŠãããŒã¿ãéžæããå¿ èŠããªããªãã1ãã«ã®ç¯çŽã«ã¯ãªããŸããã
åéããŠããåæ
補åãšãªã¢ããšã«ã³ã¹ããåé¡ããã«ã¯ãè«æ±ã·ã¹ãã ã®ããŒã¿ãããŠã³ããŒãããå¿ èŠããããŸããã€ãŸãã次ã®ããŒã¿ãåéãããã®åŸçµåããŸãã
- AWS CSVã®è«æ±æ å ±ã¯ããã¹ãŠã®è²»çšé ç®ã§AWSãçæããCSVã§ãã
- ã©ãã«ä»ãAWSãªãœãŒã¹âè«æ±CSVã§ã¿ã°ä»ãã§ãããªãœãŒã¹ã
- ã©ãã«ã®ãªããªãœãŒã¹ã¯ãã補åã©ã€ã³ãã§ãªãœãŒã¹ã®æ¶è²»ãããŒã¯ããããã«ç¹å¥ãªããŒã¿ãã€ãã©ã€ã³ãå¿ èŠãšããEBSãECSãªã©ã®ãµãŒãã¹ã§ãã
ãã®ãã¹ãŠã®ããŒã¿ã®è£œåã©ã€ã³ãç¹å®ããããããããRedshiftã§åæããããã«ããŒãã§ããŸãã
1. AWS CSVè«æ±æž
ã³ã¹ãã®åæã¯ãAWSããã®CSVãã¡ã€ã«ã®è§£æããå§ãŸããŸãã è«æ±ããŒã¿ã«ã§å¯Ÿå¿ãããªãã·ã§ã³ãæå¹ã«ã§ããŸããAmazonã¯ãæ¯æ¥S3ã«è©³çŽ°ãªè«æ±æ å ±ãå«ãCSVãã¡ã€ã«ãä¿åããŸãã
詳现ãšã¯ãéåžžã«è©³çŽ°ãªæå³ã§ãã ã¬ããŒãã®å žåçãªè¡ã¯æ¬¡ã®ãšããã§ãã
record_type | æ现 record_id | 60280491644996957290021401 product_name | Amazon DynamoDB rate_id | 0123456 subscription_id | 0123456 äŸ¡æ Œèšå®ãã©ã³ID | 0123456 usage_type | USW2-TimedStorage-ByteHrs æäœ| æšæºã¹ãã¬ãŒãž availability_zone | us-west-2 reserved_instance | N item_description | æåã®25 GBã®ç¡æGBæãè¶ ããŠäœ¿çšãããGBæããã0.25ãã« usage_start_date | 2017-02-07 03:00:00 usage_end_date | 2017-02-07 04:00:00 usage_quantity | 6e-08 blended_rate | 0.24952229400 blended_cost | 0.00000001000 unblended_rate | 0.25000000000 unblended_cost | 0.00000001000 resource_id | arnïŒawsïŒdynamodbïŒus-west-2ïŒ012345ïŒtable / a-table statement_month | 2017-02-01
ããã¯ã2æ7æ¥ã®å€3:00ãã4:00ã®éã®å¯äžã®ããŒãã«ãDynamoDBããŒã¿ããŒã¹ã«æ ŒçŽããããã®ã0.00000001ãšããé©ãã¹ãéé¡ãã€ãŸã100äžåã®1ã®ã¢ã«ãŠã³ãã§ãã éåžžãCSVã«ã¯ãã®ãããªã¬ã³ãŒããçŽ600äžä»¶å«ãŸããŠããŸãã ïŒ æ®å¿µãªããšã«ããããã®ã»ãšãã©ã¯100äžåã®1ã»ã³ããããéèŠãªéã§ã ïŒã
S3ããRedshiftã«ããŒã¿ã転éããã«ã¯ãHerokuã®awsdetailedbillingããŒã«ã䜿çšããŸãã ããã¯éå§ããã®ã«é©ããå Žæã§ãããç¹å®ã®AWSã³ã¹ãã補åã©ã€ã³ã«é¢é£ä»ããéåžžã®æ¹æ³ããããŸããã§ããïŒã€ãŸããçµ±åãŸãã¯ããŒã¿ãŠã§ã¢ããŠãžã³ã°ã«ç¹å®ã®ã€ã³ã¹ã¿ã³ã¹æéã䜿çšããŸããïŒã
ããã«ãã³ã¹ãã®çŽ60ïŒ ã¯EC2ã«ãããŸãã ããã¯ã³ã¹ãã®å€§éšåãå ããŸãããAWSãçæããCSVããã®ã¿EC2ã€ã³ã¹ã¿ã³ã¹ãšç¹å®ã®è£œåã©ã€ã³éã®æ¥ç¶ãç解ããããšã¯çµ¶å¯Ÿã«äžå¯èœã§ãã
ã€ã³ã¹ã¿ã³ã¹ã®ååã ãã§è£œåã©ã€ã³ãèå¥ã§ããªãã£ãéèŠãªçç±ããããŸãã å®éããã¹ãã§1ã€ã®ããã»ã¹ãéå§ãã代ããã«ã ECSïŒElastic Container ServiceïŒãéäžçã«äœ¿çšããŠãã¹ãã«æ°çŸã®ã³ã³ãããŒãé 眮ãããªãœãŒã¹ãå€§å¹ ã«éäžçã«äœ¿çšããŠããŸãã

æ®å¿µãªãããAmazonã¢ã«ãŠã³ãã«ã¯EC2ã€ã³ã¹ã¿ã³ã¹ã®ã³ã¹ãããå«ãŸããŠããªããããã€ã³ã¹ã¿ã³ã¹ã§å®è¡ãããŠããã³ã³ãããŒã®ã³ã¹ãã«é¢ããæ å ±ã¯ãããŸããã§ããïŒå®æçã«å®è¡ãããŠããã³ã³ãããŒã®æ°ã䜿çšããããŒã«ã®éšåãé¢äžããCPUããã³ã¡ã¢ãªãŠãããã®æ°ã
ããã«æªãããšã«ãCSVã®ã©ãã«ãã³ã³ããã®èªåã¹ã±ãŒãªã³ã°ã«é¢ããæ å ±ãåæ ãããŠããŸãã ã ãã®ããŒã¿ãåæçšã«ååŸããã«ã¯ãæ å ±ãåéããããã®ç¬èªã®ããŒã«ãäœæããå¿ èŠããããŸããã 次ã®ç« ã§ã¯ããã®ãã€ãã©ã€ã³ã®ä»çµã¿ã«ã€ããŠè©³ãã説æããŸãã
ãã ããAWS CSVãã¡ã€ã«ã¯ããµãŒãã¹ã®äœ¿çšã«é¢ããéåžžã«åªãã詳现ããŒã¿ãæäŸãããããåæã®åºç€ãšãªããŸããã ããããåœç€Ÿã®è£œåã©ã€ã³ã«æ¥ç¶ããã ãã§ãã
泚ïŒãã®åé¡ã¯ã©ãã«ãè¡ããŸããã ã€ã³ã¹ã¿ã³ã¹ã¯ããã¯ã®è«æ±ã¯ããäœã«ãéã䜿ãã®ãããšãã質åã®èŠ³ç¹ãããŸããŸãæžå¿µãããããã«ãªããŸããããã¯ãECSãKubernetesãMesosãªã©ã®ã·ã¹ãã ã䜿çšããå€æ°ã®ã€ã³ã¹ã¿ã³ã¹ã§å€ãã®ã³ã³ãããŒãèµ·åããäŒæ¥ãå¢ããŠããããã§ãã EC2ã®åã€ã³ã¹ã¿ã³ã¹ã¯ãåãç©çãµãŒããŒäžã®ä»ã®ã€ã³ã¹ã¿ã³ã¹ãšé£æºããŠåäœããXenãã€ããŒãã€ã¶ãŒã§ãããããAmazonèªäœãé·å¹Žã«ããã£ãŠãã®åé¡ãæ£ç¢ºã«çµéšããŠãããšããäºå®ã«ã¯ç®èããããŸãã
2. AWSã¿ã°ä»ããªãœãŒã¹ã®ã³ã¹ãããŒã¿
æãéèŠã§ããã«åŠçã§ããããŒã¿ã¯ãAWSã®ãã¿ã°ä»ãããªãœãŒã¹ããååŸãããŸãã
ããã©ã«ãã§ã¯ãè«æ±CSVã«ã¯ã¿ã°ãå«ãŸããŠããŸããã ãããã£ãŠãããEC2ã€ã³ã¹ã¿ã³ã¹ãŸãã¯ãã±ãããå¥ã®EC2ã€ã³ã¹ã¿ã³ã¹ãŸãã¯ãã±ãããšã©ã®ããã«åäœããããåºå¥ããããšã¯ã§ããŸããã
ãã ãã ã³ã¹ãé åžã¿ã°ã䜿çšããŠãåãŠãããã®è²»çšã®æšªã«è¡šç€ºãããã©ãã«ãã¢ã¯ãã£ãåã§ããŸã ã
ãããã®ã¿ã°ã¯ãå€ãã®AWSãªãœãŒã¹ãS3ãã±ãããDynamoDBããŒãã«ãªã©ã«ãã£ãŠå ¬åŒã«ãµããŒããããŠããŸãã CSVã§ã³ã¹ãé åžã¿ã°ã衚瀺ããã«ã¯ãAWSè«æ±ã³ã³ãœãŒã«ã§å¯Ÿå¿ãããªãã·ã§ã³ãæå¹ã«ã§ããŸãã 1æ¥çšåºŠã§ãéžæããã¿ã°ïŒ
product_area
ãéžæããïŒãã詳现ãªCSVã¬ããŒãã®å¯Ÿå¿ãããªãœãŒã¹ã®æšªã«æ°ããåãšããŠè¡šç€ºããå§ããŸãã
æé©åãè¡ã£ãŠããªãå Žåã¯ãããã«ã³ã¹ãåæ£ã¿ã°ã䜿çšããŠã€ã³ãã©ã¹ãã©ã¯ãã£ãããŒã¯ã¢ããã§ããŸãã ããã¯åºæ¬çã«ãç¡æãã®ãµãŒãã¹ã§ãããæ©èœããããã®ã€ã³ãã©ã¹ãã©ã¯ãã£ã¯å¿ èŠãããŸããã
æ©èœãã¢ã¯ãã£ãã«ããåŸã2ã€ã®ã¿ã¹ã¯ããããŸããã1ïŒæ¢åã®ã€ã³ãã©ã¹ãã©ã¯ãã£å šäœãããŒã¯ã¢ããããã 2ïŒãã¹ãŠã®æ°ãããªãœãŒã¹ãèªåçã«ã¿ã°ä»ããããããšã確èªããŸãã
æ¢åã®ã€ã³ãã©ã¹ãã©ã¯ãã£ã®ããŒã¯ã¢ãã
æ¢åã®ã€ã³ãã©ã¹ãã©ã¯ãã£ã®ã¿ã°ä»ãã¯éåžžã«ç°¡åã§ããç¹å®ã®AWS補åããšã«ãæãé«ãã³ã¹ãã®ãªãœãŒã¹ã®ãªã¹ããRedshiftã«èŠæ±ãããããã®ãªãœãŒã¹ã«ã¿ã°ãä»ããæ¹æ³ãæ瀺ãããŸã§Slackã®ãŠãŒã¶ãŒã«è¿·æããããŸãã 90ïŒ ä»¥äžã®ãªãœãŒã¹ã«ã³ã¹ãããããŠã¿ã°ãä»ããããæé ãå®äºããŸãã
ãã ãã æ°ãããªãœãŒã¹ã«ã¿ã°ãä»ããã«ã¯ãèªååãšããŒã«ãå¿ èŠã§ãã
ãã®ããã«Terraformã䜿çšããŸãã ã»ãšãã©ã®å ŽåãTerraformæ§æã¯ãAWSã³ã³ãœãŒã«ãä»ããŠè¿œå ãããã®ãšåãã³ã¹ãå²ãåœãŠã¿ã°ã®è¿œå ããµããŒãããŠããŸãã S3ãã±ããã®Terraformæ§æã®äŸã次ã«ç€ºããŸãã
resource "aws_s3_bucket" "staging_table" { bucket = "segment-tasks-to-redshift-staging-tables-prod" tags { product_area = "data-analysis" # this tag is what shows up in the billing CSV } }
Terraformã¯åºæ¬çãªæ§æãæäŸããŸãããæ°ãããªãœãŒã¹
aws_s3_bucket
Terraformãã¡ã€ã«
aws_s3_bucket
ãã
aws_s3_bucket
ã
product_area
ã¿ã°ãæ·»ä»ãããããã«ããå¿ èŠããããŸããã
幞ããªããšã«ãTerraformã®æ§æã¯HCLïŒHashicorp Configuration LanguageïŒã§èšè¿°ãããŠãããã³ã¡ã³ããä¿åããæ§æããŒãµãŒããããŸãã ããã§ããã¹ãŠã®Terraformãã¡ã€ã«ã調ã¹ãŠã
product_area
ã¿ã°ãªãã§ã¿ã°ä»ãããããªãœãŒã¹ãæ¢ãæ€èšŒé¢æ°ãäœæããŸããã
func checkItemHasTag(item *ast.ObjectItem, resources map[string]bool) error { // looking for "resource" "aws_s3_bucket" or similar t := findS3BucketDeclaration(item) tags, ok := hclchecker.GetNodeForKey(t.List, "tags") if !ok { return fmt.Errorf("aws_s3_bucket resource has no tags", resource) } t2, ok := tags.(*ast.ObjectType) if !ok { return fmt.Errorf("expected 'tags' to be an ObjectType, got %#v", tags) } productNode, ok := hclchecker.GetNodeForKey(t2.List, "product_area") if !ok { return errors.New("Could not find a 'product_area' tag for S3 resource. Be sure to tag your resource with a product_area") } }
Terraformã®æ§æãšãªããžããªã®ç¶ç¶çãªçµ±åã確ç«ãããããã®ãã§ãã¯ãè¿œå ããããã
product_area
ã¿ã°ãªãã§ã¿ã°ä»ããããªãœãŒã¹ãããå Žåããã¹ãã¯ãã¹ããŸããã
ããã¯çæ³çã§ã¯ãããŸããããã¹ãã¯çŽ°ãããæè¡çã«ã¯ãAWSã³ã³ãœãŒã«ã§ã¿ã°ãªããªãœãŒã¹ãçŽæ¥äœæããããšãã§ããŸãããã·ã¹ãã ã¯ãã®æ®µéã§ã¯ååã«æ©èœããŸãã æ°ããã€ã³ãã©ã¹ãã©ã¯ãã£ãèšè¿°ããæãç°¡åãªæ¹æ³ã¯ãTerraformã䜿çšããããšã§ãã

ã³ã¹ãé åã¿ã°ããã®ããŒã¿ã®åŠç
ãªãœãŒã¹ãããŒã¯ã¢ããããåŸããããã®ã¢ã«ãŠã³ãã£ã³ã°ã¯ç°¡åãªã¿ã¹ã¯ã§ãã
- ãªãœãŒã¹IDã
product_area
ã¿ã°ãšäžèŽãããããã«ãåãªãœãŒã¹ã®product_area
ã¿ã°ãæ€çŽ¢ããŸãã - ãã¹ãŠã®ãªãœãŒã¹ã®ã³ã¹ããåèšããŸãã
- 補åãšãªã¢ã®ã³ã¹ããè¿œå ããçµæãèŠçŽè¡šã«æžã蟌ã¿ãŸãã
SELECT sum(unblended_cost) FROM awsbilling.line_items WHERE statement_month = $1 AND product_name='Amazon DynamoDB';
AWSãµãŒãã¹ã®ã³ã¹ããåå²ããããšããå§ãããŸããã»ã°ã¡ã³ã補åã©ã€ã³çšãšAWSãµãŒãã¹çšã®2ã€ã®ããŒãã«ããããŸãã
éåžžã®AWSã®ã³ã¹ãå²ãåœãŠã¿ã°ã䜿çšããŠãçŽ35ïŒ ã®ã³ã¹ããå²ãåœãŠãŸããã
äºçŽãããã€ã³ã¹ã¿ã³ã¹ã®åæ
ãã®ã¢ãããŒãã¯ãã©ãã«ä»ãã®ã¢ã¯ã»ã¹å¯èœãªã€ã³ã¹ã¿ã³ã¹ã«é©ããŠããŸãã ãã ããå Žåã«ãã£ãŠã¯ãAWSã¯ãäºçŽãã®åæããè¡ããŸãã äºçŽã¯ãå²åŒæéã§ã®åæããšåŒãæãã«ãäžå®éã®ãªãœãŒã¹ã®å¯çšæ§ãä¿èšŒããŸãã
ç§ãã¡ã®å Žåãæšå¹Ž12æã®CSVã¢ã«ãŠã³ãããã®ããã€ãã®å€§ããªæ¯æãã¯ãä»å¹Žã®ãã¹ãŠã®æã«ããã£ãŠåé ãããå¿ èŠãããããšãããããŸããã
ãããã®è²»çšãæ£ããèæ ®ããããã«ããã®æéã®åå¥ã®ïŒãã¬ã³ããããŠããªãïŒè²»çšã®ããŒã¿ã䜿çšããããšã«ããŸããã ã¯ãšãªã¯æ¬¡ã®ããã«ãªããŸãã
select unblended_cost, usage_start_date, usage_end_date from awsbilling.line_items where start_date < '2017-04-01' and end_date > '2017-03-01' and product_name = 'Amazon DynamoDB' and resource_id = '';
ãµãã¹ã¯ãªãã·ã§ã³ã³ã¹ãã¯ã$ X0000 of DynamoDBãã®åœ¢åŒã§èšé²ãããããããªãœãŒã¹ã補åã©ã€ã³ã«èµ·å ããããšã¯ãããŸããã
代ããã«ãåãªãœãŒã¹ã®ã³ã¹ãã補åãšãªã¢ã«è¿œå ããå²åã«åŸã£ãŠãµãã¹ã¯ãªãã·ã§ã³ã³ã¹ããåé ããŸãã ããŒã¿ãŠã§ã¢ããŠã¹ãEC2è²»çšã®60ïŒ ãè²»ãããå Žåããããã®ç®çã®ããã«å å ¥è ã®60ïŒ ãåæãããããšä»®å®ããŸãã
ãããå®ç§ã§ã¯ãããŸããã ã¢ã«ãŠã³ãã®å€§éšåãåæãããååŸãããå Žåããã®ãããªé åžæŠç¥ã¯ã皌åäžã®ã€ã³ã¹ã¿ã³ã¹ã®å®è¡ã³ã¹ãã®ããããªå€æŽã«ãã£ãŠæªããããŸãã ãã®å ŽåãåãªãœãŒã¹ã®äœ¿çšã«é¢ããæ å ±ã«åºã¥ããŠçµè²»ãé åããå¿ èŠããããŸããçµè²»ãããèŠçŽããããšã¯å°é£ã§ãã
3.ã¿ã°ãªãã®AWSãªãœãŒã¹ããã®ã³ã¹ãããŒã¿
DynamoDBã€ã³ã¹ã¿ã³ã¹ãšããŒãã«ã®ããŒãã£ã·ã§ã³åå²ã¯åªããŠããŸãããä»ã®AWSãªãœãŒã¹ã¯ã³ã¹ãå ±æã¿ã°ããµããŒãããŠããŸãã ã ãããã®ãªãœãŒã¹ã§ã¯ãã³ã¹ãããŒã¿ãæ£åžžã«ååŸããŠRedshiftã«è»¢éããããã«ãRab-Goldbergã¹ã¿ã€ã«ã®åçŽãªã¯ãŒã¯ãããŒãäœæããå¿ èŠããããŸããã
ãã®å Žåã®ã¿ã°ãªããªãœãŒã¹ã®æ倧ã®2ã€ã®ã°ã«ãŒãã¯ãECSãšEBSã§ãã
ECS
ECSã·ã¹ãã ã¯ãåãµãŒãã¹ãåäœããå¿ èŠãããã³ã³ããã®æ°ã«å¿ããŠããµãŒãã¹ã®èŠæš¡ãç¶ç¶çã«å¢æžããŸãã 圌女ã¯ãŸããå€æ°ã®åã ã®ã€ã³ã¹ã¿ã³ã¹ã®åãã©ã³ã¹ãšã³ã³ãããžã®ããã±ãŒãžã³ã°ãæ åœããŠããŸãã
ECSã¯ããäºçŽãããCPUãšã¡ã¢ãªãã®æ°ã«å¿ããŠããã¹ãäžã§ã³ã³ãããå®è¡ããŸãã åãµãŒãã¹ã¯å¿ èŠãªCPUããŒãã®æ°ã瀺ããECSã¯ååãªãªãœãŒã¹ãåãããã¹ãã«æ°ããã³ã³ãããŒãé 眮ããããå¿ èŠãªãªãœãŒã¹ãè¿œå ããããã«ã€ã³ã¹ã¿ã³ã¹ã®æ°ãã¹ã±ãŒãªã³ã°ããŸãã
ãããã®ECSã¢ã¯ã·ã§ã³ã¯ããããCSVè«æ±ã¬ããŒãã«çŽæ¥åæ ãããŸããããECSã¯ãã¹ãŠã®ã€ã³ã¹ã¿ã³ã¹ã®èªåã¹ã±ãŒãªã³ã°ãå®è¡ãã責任ããããŸãã
ç°¡åã«èšããšãåã³ã³ããã䜿çšããç¹å®ã®ãã·ã³ã®ãããŒãããç解ãããã£ãã®ã§ãããCSVã¬ããŒãã§ã¯ãã€ã³ã¹ã¿ã³ã¹ããšã®ãå šäœãŠããããã®å èš³ã®ã¿ãæäŸãããŸãã
ç¹å®ã®ãµãŒãã¹ã®ã³ã¹ãã決å®ããããã«ãç¬èªã®ã¢ã«ãŽãªãºã ãéçºããŸããã
- ECSã¿ã¹ã¯ãéå§ãŸãã¯åæ¢ãããšãã«ããã¹ãŠã®ã€ãã³ãã«Cloudwatchãµãã¹ã¯ãªãã·ã§ã³ãèšå®ããŸãã
- ãã®ã€ãã³ãã«é¢é£ããããŒã¿ïŒãµãŒãã¹åãCPU /ã¡ã¢ãªäœ¿çšéãéå§ãŸãã¯åæ¢ãEC2ã€ã³ã¹ã¿ã³ã¹IDïŒãKinesis Firehoseã«éä¿¡ããŸãïŒåã ã®ã€ãã³ããèç©ããããïŒã
- Kinesis FirehoseããRedshiftã«ããŒã¿ãéä¿¡ããŸãã
ãã¹ãŠã®éå§/åæ¢/ãµã€ãºããŒã¿ãRedshiftã«å°çãããããã®ã¿ã¹ã¯ãECSã§åäœããæéïŒ120ç§ãªã©ïŒã«ããã®ãã·ã³ã§äœ¿çšããCPUãŠãããã®æ°ïŒæ倧4096-ãã®æ å ±ã¯ã€ã³ã¹ã¿ã³ã¹ã§å®è¡ãããŠããåãµãŒãã¹ã®CPUç§æ°ãèšç®ããŸãã
ã¢ã«ãŠã³ãããã®ã€ã³ã¹ã¿ã³ã¹ã®ç·ã³ã¹ãã¯ã䜿çšãããCPUç§æ°ã«å¿ããŠãµãŒãã¹éã§åå²ãããŸãã
ãããçæ³çãªæ¹æ³ã§ã¯ãããŸããã EC2ã€ã³ã¹ã¿ã³ã¹ã¯åžžã«100ïŒ ã®é»åã§åäœããããã§ã¯ãããŸããããéå°ã¯çŸåšããã®ã€ã³ã¹ã¿ã³ã¹ã§åäœãããã¹ãŠã®ãµãŒãã¹ã«åé ãããŠããŸãã ããã¯ãéå°è²»çšã®æ£ããåé ãŸãã¯èª€ã£ãåé ã§ããå¯èœæ§ããããŸãã ãããïŒãããŠãããã§ãã®èšäºã®äžè¬çãªãããã¯ãèŠã€ããããšãã§ããŸãïŒããã§ååã§ãã

ããã«ãåECSãµãŒãã¹ã察å¿ãã補åã©ã€ã³ãšé¢é£ä»ããå¿ èŠããããŸãã ãã ããECS ã¯ã³ã¹ãå ±æã¿ã°ããµããŒãããŠããªããããAWSã§ããããããŒã¯ããããšã¯ã§ããŸããã
代ããã«ãåECSãµãŒãã¹ã®Terraformã¢ãžã¥ãŒã«ã«
product_area
ããŒãè¿œå ããŸãã ãã®ããŒã¯ãAWSã«éä¿¡ãããã¡ã¿ããŒã¿ã«ã¯ã€ãªãããŸãããããã¹ãŠã®ãµãŒãã¹ã®
product_area
ããŒãèªã¿åãã¹ã¯ãªãããçæããŸãã
ãã®ã¹ã¯ãªããã¯ãæ°ããããŒã¿ãéä¿¡ãããã³ã«ãDynamoDBã®ã¡ã€ã³ãã©ã³ãã§ãµãŒãã¹åãšè£œåã®æ¹åã®base64ãšã³ã³ãŒãããããå ¬éããŸãã

æåŸã«ããã¹ãã§ã¯ã補åã©ã€ã³ã§ã¿ã°ä»ããããåæ°ãããµãŒãã¹ããã§ãã¯ããŸãã
Ebs
Elastic Block StorageïŒEBSïŒãã¢ã«ãŠã³ãã®å€§éšåãå ããŠããŸãã EBSããªã¥ãŒã ã¯éåžžEC2ã€ã³ã¹ã¿ã³ã¹ã«æ¥ç¶ãããŠãããäŒèšäžãEBSããªã¥ãŒã ã®ã³ã¹ããšå¯Ÿå¿ããEC2ã€ã³ã¹ã¿ã³ã¹ãèæ ®ããããšã¯çã«ããªã£ãŠããŸãã ãã ããAWS CSVè«æ±ã§ã¯ã ã©ã® EBSããªã¥ãŒã ãã©ã®ã€ã³ã¹ã¿ã³ã¹ã«æ¥ç¶ãããŠãããã¯è¡šç€ºãããŸããã

ãããè¡ãããã«ãåã³Cloudwatchã䜿çšããŸããããããªã¥ãŒã æ¥ç¶ãããããªã¥ãŒã åæããªã©ã®ã€ãã³ãã«ãµãã¹ã¯ã©ã€ãããDynamoDBããŒãã«ã«EBS => EC2æ¥ç¶ãç»é²ããŸããã
次ã«ãECSã®ã³ã¹ããèæ ®ããåã«ãEBSããªã¥ãŒã ã®ã³ã¹ãã察å¿ããEC2ã€ã³ã¹ã¿ã³ã¹ã®ã³ã¹ãã«è¿œå ããŸãã
ã¢ã«ãŠã³ãéã§ããŒã¿ãçµåãã
ãããŸã§ãåäžã®AWSã¢ã«ãŠã³ãã®ã³ã³ããã¹ãã§ãã¹ãŠã®è²»çšã«ã€ããŠèª¬æããŠããŸããã ãããå®éã«ã¯ãããã¯ç°ãªãç©çAWSã¢ã«ãŠã³ãéã§å ±æãããå®éã®AWSèšå®ãåæ ããŠããŸããã

éçšã¢ã«ãŠã³ãïŒOpsïŒã䜿çšããŠããã¹ãŠã®ã¢ã«ãŠã³ãã®ããŒã¿ãšè«æ±ãçµ±åããã ãã§ãªãããšã³ãžãã¢ãæ¬çªç°å¢ã§å€æŽãå®è£ ããããã®åäžã®ã¢ã¯ã»ã¹ãã€ã³ããæäŸããŸãã Stageã¹ããŒãžãProductionã¹ããŒãžããåé¢ããŸããããã«ãããDynamoDBããŒãã«ãåé€ãããªã©ã®APIåŒã³åºããé©åãªãã§ãã¯ã§å®å šã«åŠçãããããšã確èªã§ããŸãã
Prodã¢ã«ãŠã³ãã¯ãããã®è²»çšã¢ã«ãŠã³ãã®äžã§æ¯é çã§ãããã¹ããŒãžã¢ã«ãŠã³ãã®è²»çšãAWSã¢ã«ãŠã³ãå šäœã®ããªãã®éšåãå ããŠããŸãã
ã¹ããŒãžã¢ã«ãŠã³ãããå®çšŒåç°å¢ã®Redshiftã¯ã©ã¹ã¿ãŒã«ECSãµãŒãã¹ã«é¢ããããŒã¿ãæžã蟌ãå¿ èŠãããå Žåãåé¡ãå§ãŸããŸãã
ãã¢ã«ãŠã³ãéããèšé²ã§ããããã«ããã«ã¯ãCloudwatchãµãã¹ã¯ãªãã·ã§ã³ããã»ããµãFirehoseïŒECSã®å ŽåïŒãŸãã¯DynamoDBïŒEBSã®å ŽåïŒã§ã®èšé²ã®ããã«æ¬çªç°å¢ã§ã®åœ¹å²ãåŒãåããå¿ èŠããããŸãã Stageã¢ã«ãŠã³ãïŒsts.AssumeRoleïŒããã³Prodã¢ã«ãŠã³ãã«é©åãªæ©èœã®é©åãªã¢ã¯ã»ã¹èš±å¯ãè¿œå ããå¿ èŠããããããããã¯ç°¡åã§ã¯ãããŸããããšã©ãŒããããšãã¢ã¯ã»ã¹èš±å¯ãæ··ä¹±ããŸãã

ç§ãã¡ã«ãšã£ãŠãããã¯ã¢ã«ãŠã³ãã£ã³ã°ã³ãŒããStageã¢ã«ãŠã³ãã§æ©èœããããã¹ãŠã®æ å ±ãProdã¢ã«ãŠã³ãã®ããŒã¿ããŒã¹ã«å ¥åãããããšãæå³ããŸãã
åãããŒã¿ããµãã¹ã¯ã©ã€ãããStageã¢ã«ãŠã³ãã«2ã€ç®ã®ãµãŒãã¹ãè¿œå ã§ããŸãããããã¯èšé²ãããŸãããããã®å ŽåãStageã¢ã«ãŠã³ãã£ã³ã°ã³ãŒãã«ã©ã³ãã ãªåé¡ãçºçããå¯èœæ§ããããšå€æããŸããã
çµ±èšåºå
æåŸã«ãé©åãªããŒã¿åæã®ããã®ãã¹ãŠããããŸãã
- CSVã§ãªãœãŒã¹ãããŒã¯ã¢ããããŸããã
- åECSã€ãã³ããéå§ããã³åæ¢ãããšãã®ããŒã¿ã
- ECSãµãŒãã¹åãé¢é£è£œåã©ã€ã³ã«ãã€ã³ãããŸãã
- EBSããªã¥ãŒã ãæ¥ç¶æžã¿ã€ã³ã¹ã¿ã³ã¹ã«ãã€ã³ãããŸãã
ããããã¹ãŠåæããŒã ã«æäŸããããã«ãAWSããŒã¿ã売ã蟌ã¿ãŸããã åAWSãµãŒãã¹ã«ã€ããŠããã®AWSãµãŒãã¹ã®ã»ã°ã¡ã³ã補åã©ã€ã³ãšãã®ã³ã¹ãããŸãšããŸããã
ãã®ããŒã¿ã¯ã3ã€ã®ç°ãªãããŒãã«ã«è¡šç€ºãããŸãã
- ç¹å®ã®æã®åECSãµãŒãã¹ã®ç·è²»çšã
- ç¹å®ã®æã®å補åã©ã€ã³ã®ç·ã³ã¹ãã
- ç¹å®ã®æã®ïŒAWSãµãŒãã¹ãã»ã°ã¡ã³ã補åã©ã€ã³ïŒã®ç·ã³ã¹ãã ããšãã°ããããŒã¿ãŠã§ã¢ããŠãžã³ã°ã¯å æDynamoDBã«1,000ãã«ãè²»ãããŸãããã
åã ã®è£œåã©ã€ã³ã®ç·ã³ã¹ãã¯æ¬¡ã®ããã«ãªããŸãã
æ| product_area | cost_cents -------------------------------------- 2017-03-01 | çµ±å| 500 2017-03-01 | å庫| 783
ãŸããã»ã°ã¡ã³ãã®è£œåã©ã€ã³ã«é¢é£ããAWSãµãŒãã¹ã³ã¹ãã¯æ¬¡ã®ãšããã§ãã
æ| product_area | aws_product | cost_cents -------------------------------------------------- - 2017-03-01 | çµ±å| ec2 | 333 2017-03-01 | çµ±å| ãã€ãã¢ãã| 167 2017-03-01 | å庫| èµ€æ¹å移| 783
ãããã®åããŒãã«ã«ã¯ãåæã®åèšãå«ãåèšããŒãã«ãšãçŸåšã®æã®ããŒã¿ãæ¯æ¥æŽæ°ããè¿œå ã®ãµããªãŒããŒãã«ããããŸãã ããããããŒãã«ã®äžæã®èå¥åã¯åãã¹ã«å¯Ÿå¿ããŠããããããã®ãã¹ã®ãã¹ãŠã®è¡ãèŠã€ããããšã§AWSã¹ã³ã¢ãçµ±åã§ããŸãã
çµæãšããŠåŸãããããŒã¿ã¯ãé«ã¬ãã«ã®ã¡ããªãã¯ãšçµå¶é£ãžã®å ±åã«äœ¿çšãããé»éã®ãçå®ã®æºããšããŠå¹æçã«æ©èœããŸãã ãµããªãŒããŒãã«ã¯ã1ãæéã®ã©ã³ãã³ã°ã³ã¹ããç£èŠããããã«äœ¿çšãããŸãã
泚ïŒAWSã¯ãææ«ã®ãããæ°æ¥åŸã«ãæçµãè«æ±æžãçºè¡ãããããææ«ã«è«æ±ã¬ã³ãŒããæçµãšããŠããŒã¯ããããžãã¯ã¯æ£ãããããŸããã CSVãã¡ã€ã«ã®
invoice_id
ãã£ãŒã«ãã«ãæšå®ããšããåèªã§ã¯ãªãæŽæ°ã衚瀺ããããšãã«ãæçµçãªAmazonã¹ã³ã¢ãèŠã€ããããšãã§ããŸãã
æåŸã®ãã³ã
çµäºããåã«ãããã»ã¹å šäœã§ãå°ãã®æºåãšç¥èã§å€ãã®æéãç¯çŽã§ããå Žæãããããšã«æ°ä»ããŸããã ãœãŒããªãã®å Žåããããã®å Žæã¯æ¬¡ã®ãšããã§ãã
- ããŒã¿ãéçŽããããããå Žæããå¥ã®å Žæã«ã³ããŒãããããã¹ã¯ãªãããéçºè ã®æã«å±ãããšã¯ãã£ãã«ãããŸããããŸãã圌ãã®äœæ¥ã¯ããç£èŠãããŠããŸããã ããšãã°ãã¹ã¯ãªããã®1ã€ã¯S3ãã±ããããå¥ã®S3ãã±ããã«CSVè«æ±ããŒã¿ãã³ããŒããŸããããCSVãã¡ã€ã«ãååã«å€§ãããªã£ããšãã«Lambdaãã³ãã©ãŒãã³ããŒäžã«ååãªã¡ã¢ãªãæã£ãŠããªãã£ããããæ¯æ27ã28æ¥ã«ãšã©ãŒã§ã¯ã©ãã·ã¥ããŸããã RedshiftããŒã¿ããŒã¹ã«ã¯å€§éã®ããŒã¿ããããæ¯æãã£ãšããããããã«èŠãããããããã«æ°ä»ããŸã§ã«ãã°ããæéãããããŸããã ãã以éãLambdaé¢æ°ã®ç£èŠãè¿œå ããŠãã¹ã¯ãªããããšã©ãŒãªãã§å®è¡ãããããã«ããŸããã
- ãããã®ã¹ã¯ãªãããé©åã«ææžåãããŠããããšã確èªããŠãã ãããç¹ã«ãããããã©ã®ããã«é¢äžããã©ã®ãããªæ§æãå¿ èŠã§ãããã«ã€ããŠã®æ å ±ãå¿ èŠã§ãã ãããã®ã¹ã¯ãªãããèšåãããŠããä»ã®å Žæã®ãœãŒã¹ã³ãŒããžã®ãªã³ã¯ã ããšãã°ãS3ãã±ããããããŒã¿ããªã¯ãšã¹ããããã¹ãŠã®å Žæã§ããã±ããã«ããŒã¿ãæžã蟌ãã¹ã¯ãªãããžã®ãªã³ã¯ãé 眮ããŸãã ãŸããS3ãã±ããã®ã«ãŒããã£ã¬ã¯ããªã«READMEãã¡ã€ã«ãæžã蟌ãããšãæ€èšããŠãã ããã
- Redshiftãªã¯ãšã¹ãã¯ãæé©åããã«éåžžã«äœéã§å®è¡ã§ããŸãã äŒç€Ÿã®Redshiftã¹ãã·ã£ãªã¹ãã«çžè«ããRedshiftã§æ°ããããŒãã«ãäœæããåã«å¿ èŠãªã¯ãšãªãæ€èšããŠãã ããã ãã®å ŽåãCSVè«æ±ããŒãã«ã«æ£ãããœãŒãããŒããããŸããã§ããã ããŒãã«ãäœæããåŸã䞊ã¹æ¿ãããŒãè¿œå ããããšã¯ã§ããªããªããŸãã®ã§ãäºåã«ãããåŠçããªãå Žåã¯ãæ£ããããŒã§2çªç®ã®ããŒãã«ãäœæããããã«æžã蟌ã¿æäœã転éããŠããããã¹ãŠã®ããŒã¿ãã³ããŒããå¿ èŠããããŸãã
- æ£ãããœãŒãããŒã䜿çšãããšãããããããŒãã«ã®ãã¹äžã®ã¯ãšãªã¹ããŒãžã®å®è¡æéãçŽ7åãã10ã30ç§ã«ççž®ãããŸããã
- æåã¯ãã¹ã±ãžã¥ãŒã«ã«åŸã£ãŠããããããŒãã«ãæŽæ°ããã¹ã¯ãªãããå®è¡ããããšãèšç»ããŸãããCloudwatchã¯1æ¥ã«äœåºŠãAWS Lambdaé¢æ°ãå®è¡ã§ããŸãã ãã ããããã»ãŒãžãµã€ãºã¯äžè²«ããŠãããïŒç¹ã«Redshiftã«ãšã³ããªãå«ãŸããŠããå ŽåïŒãæ倧Lambdaã¿ã€ã ã¢ãŠããè¶ ããŠããããã代ããã«ECSãµãŒãã¹ã«è»¢éããŸããã
- åœåãããããããŒãã«æŽæ°ã³ãŒãã«ã¯JavaScriptãéžæããŸãããããã¯Lambdaã§åäœããåœç€Ÿã®ã»ãšãã©ã®ã¹ã¯ãªãããJavaScriptã§èšè¿°ãããŠããããã§ãã ECSã«åãæ¿ããå¿ èŠãããããšãããã£ãŠããå Žåã¯ã64ãããæ°ã®è¿œå ãäœæ¥ã®äžŠååããã³ãã£ã³ã»ã«ã®ãµããŒãã匷åãããå¥ã®èšèªãéžæããŸãã
- Redshiftã§æ°ããããŒã¿ã®æžã蟌ã¿ãéå§ãããã³ã«ãããŒã¿ãå€æŽããïŒããšãã°ãæ°ããåãè¿œå ããïŒããããŒã¿åæã«é¢ããæŽåæ§ãšã©ãŒãä¿®æ£ããå€æŽã«é¢ããæ¥ä»ãšæ å ±ãèšèŒããREADMEãã¡ã€ã«ã«ã¡ã¢ãè¿œå ããŸãã ããã¯ãããŒã¿åæããŒã ã®ã¹ã¿ããã«ãšã£ãŠéåžžã«åœ¹ç«ã¡ãŸãã
- ãã¬ã³ããããè²»çšã¯ããã®ã¿ã€ãã®åæã«ã¯ããŸã圹ã«ç«ã¡ãŸãã-ãã¬ã³ããããŠããªãè²»çšã«åºå·ããŸãã ç¹å®ã®ãªãœãŒã¹ã«å¯ŸããŠAWSãè«æ±ããéé¡ãæ£ç¢ºã«ç€ºããŸãã
- CSVã¬ããŒãã«ã¯8è¡ãŸãã¯9è¡ããããAmazonã¯ãµãŒãã¹ã®ååã瀺ããŠããŸããã ãããã¯è«æ±æžã®åèšéé¡ãè¡šããŸãããç¹å®ã®æã®æ··åãããŠããªãè²»çšãèŠçŽããè©Šã¿ãæ念ããŸãã ãããã®æ°å€ãè²»çšã®åèšããé€å€ãããŠããããšã確èªããŠãã ããã
ããã ã©ã€ã³
è¯ãAWSã¹ã³ã¢ãååŸããã®ã¯ç°¡åã§ã¯ãããŸããã ããã«ã¯ãããŒã«ã®éçºãšèšå®ãããã³AWSã§ã®é«äŸ¡ãªãªãœãŒã¹ã®ç¹å®ã®äž¡æ¹ã®ããã«ã倧éã®äœæ¥ãå¿ èŠã§ãã
ç§ãã¡ãéæããæãéèŠãªåå©ã¯ãå®æçãªã1åéãã®åæãã§ã¯ãªããã³ã¹ãã®åçŽãªç¶ç¶çäºæž¬ã®å¯èœæ§ã§ãã
ãããè¡ãããã«ãããŒã¿åéå šäœãèªååããTerraformãšç¶ç¶çã€ã³ãã°ã¬ãŒã·ã§ã³ã·ã¹ãã ã«ã¿ã°ãµããŒããå®è£ ããéçºããŒã ã®ãã¹ãŠã®ã¡ã³ããŒã«ã€ã³ãã©ã¹ãã©ã¯ãã£ã«æ£ããã¿ã°ãä»ããæ¹æ³ã説æããŸããã
ãã¹ãŠã®ããŒã¿ã¯PDFã®èªéã§ã¯ãããŸããããRedshiftã§ç¶ç¶çã«æŽæ°ãããŸãã æ°ãã質åã«çããŠæ°ããã¬ããŒããçæããå Žåã¯ãSQLã¯ãšãªã䜿çšããŠå³åº§ã«çµæãååŸããŸãã
ããã«ããã¹ãŠã®ããŒã¿ãExcel圢åŒã«ãšã¯ã¹ããŒããããããæ°ããã¯ã©ã€ã¢ã³ãã«ãããè²»çšãæ£ç¢ºã«èšç®ã§ããŸãã ãŸããç¹å®ã®ãµãŒãã¹ã補ååéã«å€é¡ã®è³éãçªç¶æµå ¥ãå§ãããã©ããã確èªã§ããŸããããã¯ãäºå®å€ã®è²»çšãäŒç€Ÿã®è²¡æ¿ç¶æ ã«éããåã«èŠãããŸãã
ãã®èšäºã¯ã客æ§ã®ã€ã³ãã©ã¹ãã©ã¯ãã£ãšå®å šã«ã¯äžèŽããŠããŸãããããã®ã¹ããŒãªãŒãã客æ§ã®ããžãã¹ã®æé·ã«åãããŠã³ã¹ããããããç解ãã管çããã®ã«åœ¹ç«ã€ããšãé¡ã£ãŠããŸãã