33 Matching Annotations
  1. Sep 2026
    1. Alternatives to MinIO for single-node local S3
      • Context & Motivation:

        • Following the MinIO company's decision to shift away from maintaining the open-source release, developers and demo creators needed alternative local S3 emulators.
        • The author's criteria focused strictly on single-node, local testing/demo environments: ready-to-use Docker images, S3 compatibility, open-source licensing (per OSI), simple configuration, and active community/backing.
        • Testing baseline used a minimal Docker Compose setup: DuckDB reading and writing Apache Iceberg data backed by local S3 storage.
      • Evaluated Solutions:

        • S3Proxy:
          • Lightweight, highly compatible, and very easy to set up (👍👍).
          • Noted downside: Depends on Apache jclouds, which was retired to the Apache Attic in mid-2025.
        • RustFS:
          • Easy setup with an included GUI and strong S3 compliance (👍👍).
          • Drawbacks: Currently in alpha release and recently experienced a notable security vulnerability.
        • SeaweedFS:
          • Mature open-source project with solid S3 compatibility and a built-in UI (👍).
          • Slight initial hurdle regarding auth config files in Docker Compose, though the maintainer promptly pushed changes to streamline single-node setup.
        • Zenko CloudServer:
          • Formerly known as S3 Server (by Scality); straightforward replacement (👍).
          • Downsides include confusing branding across Scality/Zenko/CloudServer and outdated Docker Hub images referenced in documentation.
        • Garage:
          • Lightweight distributed-focused object store with an AGPL license.
          • Steep setup curve for simple single-node demos due to custom initialization, mandatory key ID format constraints, and external TOML configuration requirements (😬).
        • Apache Ozone & Ceph Object Gateway:
          • Deemed too heavyweight and complex for lightweight single-node demo stacks (Ozone required at least four nodes; Ceph required extensive setup).
      • Author's Takeaway:

        • Recommended choices for local demos: SeaweedFS and S3Proxy (with RustFS as a potential future candidate once matured).
        • Reminded users to weigh project governance (foundation vs. single maintainer) and bus factor when selecting tools.

      Hacker News Discussion

      • MinIO Fork (pgsty/silo):
        • A prominent fork created by Ruohang Feng (originally pgsty/minio, renamed to pgsty/silo) was highlighted by multiple commenters as the most seamless drop-in replacement that requires zero config or code changes for existing MinIO users.
      • VersityGW as a Popular Alternative:
        • Several users strongly recommended versitygw for homelabs and local testing, highlighting its stability and ability to map S3 buckets directly to standard POSIX directories without complex overhead.
      • Garage Experience & Caveats:
        • Commenters noted that Garage v2.3.0 added a --single-node --default-bucket flag that significantly simplifies single-node setups compared to earlier versions.
        • However, technical drawbacks were emphasized: Garage relies on eventual consistency and lacks strict conditional PUT semantics, making it unsuitable for workloads like Apache Iceberg or Litestream-style lock guarantees.
      • SeaweedFS Community Input:
        • The SeaweedFS maintainer actively participated in the thread, clarifying performance benchmarks, welcoming issue reports for concurrency or documentation gaps, and pointing to updated documentation and broad S3 feature coverage.
      • Other Noteworthy Mentions:
        • HS5: A lightweight object store designed specifically for local S3 simulation with integrated DuckDB support and strict ETag / conditional request compliance.
        • Filestash S3 Gateway: Mentioned as a flexible proxy translating S3 requests across diverse backend storages (SFTP, SMB, NFS).
        • Incus: Noted that the LXD successor project recently adopted its own native S3 service to drop MinIO dependencies.
  2. Jan 2026
  3. Oct 2025
  4. Nov 2024
    1. Deploying Machine Learning Models with Flask and AWS Lambda: A Complete Guide

      In essence, this article is about:

      1) Training a sample model and uploading it to an S3 bucket:

      ```python from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression import joblib

      Load the Iris dataset

      iris = load_iris() X, y = iris.data, iris.target

      Split the data into training and testing sets

      X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

      Train the logistic regression model

      model = LogisticRegression(max_iter=200) model.fit(X_train, y_train)

      Save the trained model to a file

      joblib.dump(model, 'model.pkl') ```

      1. Creating a sample Zappa config, because AWS Lambda doesn’t natively support Flask, we need to use Zappa, a tool that helps deploy WSGI applications (like Flask) to AWS Lambda:

      ```json { "dev": { "app_function": "app.app", "exclude": [ "boto3", "dateutil", "botocore", "s3transfer", "concurrent" ], "profile_name": null, "project_name": "flask-test-app", "runtime": "python3.10", "s3_bucket": "zappa-31096o41b" },

      "production": {
          "app_function": "app.app",
          "exclude": [
              "boto3",
              "dateutil",
              "botocore",
              "s3transfer",
              "concurrent"
          ],
          "profile_name": null,
          "project_name": "flask-test-app",
          "runtime": "python3.10",
          "s3_bucket": "zappa-31096o41b"
      }
      

      } ```

      1. Writing a sample Flask app:

      ```python import boto3 import joblib import os

      Initialize the Flask app

      app = Flask(name)

      S3 client to download the model

      s3 = boto3.client('s3')

      Download the model from S3 when the app starts

      s3.download_file('your-s3-bucket-name', 'model.pkl', '/tmp/model.pkl') model = joblib.load('/tmp/model.pkl')

      @app.route('/predict', methods=['POST']) def predict(): # Get the data from the POST request data = request.get_json(force=True)

      # Convert the data into a numpy array
      input_data = np.array(data['input']).reshape(1, -1)
      
      # Make a prediction using the model
      prediction = model.predict(input_data)
      
      # Return the prediction as a JSON response
      return jsonify({'prediction': int(prediction[0])})
      

      if name == 'main': app.run(debug=True) ```

      1. Deploying this app to production (to AWS):

      bash zappa deploy production

      and later eventually updating it:

      bash zappa update production

      1. We should get a URL like this:

      https://xyz123.execute-api.us-east-1.amazonaws.com/production

      which we can query:

      curl -X POST -H "Content-Type: application/json" -d '{"input": [5.1, 3.5, 1.4, 0.2]}' https://xyz123.execute-api.us-east-1.amazonaws.com/production/predict

  5. Apr 2024
    1. Lesson 1: Anyone who knows the name of any of your S3 buckets can ramp up your AWS bill as they like.

      The author was charged over $1300 after two days of using an S3 bucket, because some OS tool stored a default bucket name in the config, which was the same as his bucket name.

      Luckily, after everything AWS made an exception and he did not have to pay the bill.

  6. Nov 2023
  7. Nov 2021
    1. Saving Your Wallet With Lifecycle Rules Of course, storing multiple copies of objects uses way more space, especially if you’re frequently overwriting data. You probably don’t need to store these old versions for the rest of eternity, so you can do your wallet a favor by setting up a Lifecycle rule that will remove the old versions after some time. Under Management > Life Cycle Configuration, add a new rule. The two options available are moving old objects to an infrequent access tier, or deleting them permanently after
    1. S3 object versioning Many of the strategies to be discussed for data durability require S3 object versioning to be enabled for the bucket (this includes S3 object locks and replication policies). With object versioning, anytime an object is modified, it results in a new version, and when the object is deleted, it only results in the object being given a delete marker. This allows an object to be recovered if it has been overwritten or marked for deletion. However, it is still possible for someone with sufficient privileges to permanently delete all objects and their versions, so this alone is not sufficient. When using object versioning, deleting old versions permanently is done with the call s3:DeleteObjectVersion, as opposed to the usual s3:DeleteObject, which means that you can apply least privilege restrictions to deny someone from deleting the old versions. This can help mitigate some issues, but you should still do more to ensure data durability. Life cycle policies Old versions of objects will stick around forever, and each version is an entire object, not a diff of the previous version. So if you have a 100MB file that you change frequently, you’ll have many copies of this entire file. AWS acknowledges in the documentation “you might have one or more objects in the bucket for which there are millions of versions”. In order to reduce the number of old versions, you use lifecycle policies. Audit tip: It should be a considered a misconfiguration if you have object versioning enabled and no lifecycle policy on the bucket. Every versioned S3 bucket should have a `NoncurrentVersionExpiration` lifecycle policy to eventually remove objects that are no longer the latest version. For data durability, you may wish to set this to 30 days. If this data is being backed up, you may wish to set this to as little as one day on the primary data and 30 days on the backup. If you are constantly updating the same objects multiple times per day, you may need a different solution to avoid unwanted costs. Audit tip: In 2019, I audited the AWS IAM managed policies and found some issues, including what I called Resource policy privilege escalation. In a handful of cases AWS had attempted to create limited policies that did not allow `s3:Delete*`, but still allowed some form of `s3:Put*`. The danger here is the ability to call `s3:PutBucketPolicy` in order to grant an external account full access to an S3 bucket to delete the objects and versions within it, or `s3:PutLifecycleConfiguration` with an expiration of 1 day for all objects which will delete all objects and their versions in the bucket. Storage classes With lifecycle policies, you have the ability to transition objects to less expensive storage classes. Be aware that there are many constraints, specifically around the size of the object and how long you have to keep it before transitioning or deleting it. Objects in the S3 Standard storage class must be kept there for at least 30 days until they can be transitioned. Further, once an object is in the S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA, those objects must be kept there for 30 days before deletion. Objects in Glacier must be kept for 90 days before deleting, and objects in Glacier Deep Archive must be kept for 180 days. So if you had plans of immediately transitioning all non-current object versions to Glacier Deep Archive to save money, and then deleting them after 30 days, you will not be able to.
  8. Oct 2021
    1. So, while DELETE operations are free, LIST operations (to get a list of objects) are not free (~$.005 per 1000 requests, varying a bit by region).

      Deleting buckets on S3 is not free. If you use either Web Console or AWS CLI, it will execute the LIST call per 1000 objects

  9. Sep 2020
  10. May 2020
    1. In general, bucket owners pay for all Amazon S3 storage and data transfer costs associated with their bucket. A bucket owner, however, can configure a bucket to be a Requester Pays bucket. With Requester Pays buckets, the requester instead of the bucket owner pays the cost of the request and the data download from the bucket. The bucket owner always pays the cost of storing data.

      Request Pays

    1. Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

      event notification of s3 might take minutes

      BTW,

      cloud watch does not support s3, but cloud trail does

  11. Aug 2019
  12. Aug 2018
  13. Feb 2018
  14. Dec 2017
  15. Oct 2017
  16. Jun 2017
  17. May 2017
  18. Mar 2017
  19. May 2016
    1. . Is art about making up new things or about transforming the raw material that's out there? Cutting, pasting, sampling, remixing and mashing up have become mainstream modes of cultural expression, and fan fiction is part of that. It challenges just about everything we thought we knew about art and creativity.

      Not really. Art has always been about reacting (in some part) to the works that have come before it.