http-tar-streamer is a simple HTTP server that allows you to stream tar archives of directories over HTTP. It supports both uncompressed and gzip-compressed tar archives.

• Streams tar archives of directories over HTTP, without requiring any extra space on server
• Uses minimal resources, with memory consumption under 10MB
• Supports both uncompressed and gzip-compressed tar archives
• Provides a simple web interface that displays a list of directories in the current working directory when you navigate to the root URL “https://github.com/”
• Allows you to download a tar archive of any directory by navigating to its URL with a .tar or .tar.gz extension
• Cowardly refuses to serve files if the filename contains any separator like “https://github.com/” to prevent directory traversal attacks

To use http-tar-streamer, you can either run it directly from the command line or build it as a standalone binary.

To run http-tar-streamer from the command line, use the following command:

go run main.go

This will start the server on port 8080 and serve the current working directory.

To build http-tar-streamer as a standalone binary, use the following command:

go build -ldflags "-s -w" -o bin/http-tar-streamer main.go

This will create a standalone binary named http-tar-streamer in the current working directory. You can then run the binary using the following command:

./bin/http-tar-streamer

This will start the server on port 8080 and serve the current working directory.

To download a tar archive of a directory, navigate to the URL for that directory with a .tar or .tar.gz extension. For example, if you have a directory named mydir in the current working directory, you can download a tar archive of that directory using the following URLs:

• http://localhost:8080/mydir.tar (uncompressed tar archive)
• http://localhost:8080/mydir.tar.gz (gzip-compressed tar archive)

You can also use curl to download the tar archive and get the download speed. For example:

curl -o /dev/null -s -w %{speed_download} http://localhost:8080/mydir.tar

This will download the tar archive to /dev/null and print the download speed in bytes per second.

http-tar-streamer does not support streaming tar archives of individual files, only directories.

git clone https://github.com/tobiasrausch/ATACseq.git

cd ATACseq

make all

If one of the above commands fail your operating system probably lacks some build essentials. These are usually pre-installed but if you lack them you need to install these. For instance, for Ubuntu this would require:

apt-get install build-essential g++ git wget unzip

To annotate motifs and estimate TSS enrichments some simple scripts are included in this repository to download these databases.

cd bed/ && Rscript promoter.R && cd ..

cd motif/ && ./downloadMotifs.sh && cd ..

./src/atac.sh <hg38|hg19|mm10> <read1.fq.gz> <read2.fq.gz> <genome.fa> <output prefix>

The pipeline produces at various steps JSON QC files (*.json.gz). You can upload and interactively browse these files at https://gear-genomics.embl.de/alfred/. In addition, the pipeline produces a succinct QC file for each sample. If you have multiple output folders (one for each ATAC-Seq sample) you can simply concatenate the QC metrics of each sample.

head -n 1 ./*/*.key.metrics | grep "TssEnrichment" | uniq > summary.tsv

cat ./*/*.key.metrics | grep -v "TssEnrichment" >> summary.tsv

To plot the distribution for all QC parameters.

Rscript R/metrics.R summary.tsv

The ATAC-Seq pipeline produces various output files.

• Bowtie BAM alignment files filtered for duplicates and mitochondrial reads.
• Quality control output files from alfred, samtools, FastQC and cutadapt adapter filter metrics.
• Macs peak calling files and IDR filtered peak lists.
• Succinct browser tracks in bedGraph format and IGV’s tdf format.
• Footprint track of nucleosome positions and/or transcription factor bound DNA.
• Homer motif finding results.

Merge peaks across samples and create a raw count matrix.

ls ./Sample1/Sample1.peaks ./Sample2/Sample2.peaks ./SampleN/SampleN.peaks > peaks.lst

ls ./Sample1/Sample1.bam ./Sample2/Sample2.bam ./SampleN/SampleN.bam > bams.lst

./src/count.sh hg19 peaks.lst bams.lst <output prefix>

To call differential peaks on a count matrix for TSS peaks, called counts.tss.gz, using DESeq2 we first need to create a file with sample level information (sample.info). For instance, if you have 2 replicates per condition:

echo -e "name\tcondition" > sample.info

zcat counts.tss.gz | head -n 1 | cut -f 5- | tr '\t' '\n' | sed 's/.final$//' | awk '{print $0"\t"int((NR-1)/2);}' >> sample.info

Rscript R/dpeaks.R counts.tss.gz sample.info

Peaks can of course be intersected with enhancer or conserved element tracks, i.e.:

cd tracks/ && downloadTracks.sh

bedtools intersect -a ./Sample2/Sample2.peaks -b tracks/conserved.bed

There is a basic Rscript available for plotting peak densities.

Rscript R/karyoplot.R input.peaks

Tobias Rausch, Markus Hsi-Yang Fritz, Jan O Korbel, Vladimir Benes.
Alfred: Interactive multi-sample BAM alignment statistics, feature counting and feature annotation for long- and short-read sequencing.
Bioinformatics. 2018 Dec 6.

B Erarslan, JB Kunz, T Rausch, P Richter-Pechanska et al.
Chromatin accessibility landscape of pediatric T‐lymphoblastic leukemia and human T‐cell precursors
EMBO Mol Med (2020)

This ATAC-Seq pipeline is distributed under the BSD 3-Clause license.

This codebase was created to demonstrate a backend application built with Java 11 + Spring Boot 3 including CRUD operations, authentication, routing and more.

This app helps users to rate a hotel services based on their experience. Once user is logged with correct email and password, he can post and see the reviews given by other users.

This app mainly contains 3 microservices-

1. UserServices: This service contains information about the user like his id, name and email.It uses MySQL as a database.

2. HotelServices: This service contains information about the hotel like his id, name, ratings given by users and feedback provided.It uses Oracle as a database.

3. RatingServices: This service provides ratings given to hotels based on userID and hotelId .It uses MongoDB as a database.

Apart from these services, Okta has been used to provide security to microservices.

Taking a closer look at the project structure, the main code of the application is located in the src/main/java directory. Additionally, configuration files and such can be found in the src/main/resources directory.

The core logic of the application is organized as follows:

~Controller: Processes HTTP requests, calls business logic, and generates responses.

~Service: Implements business logic and interacts with the database through Repositories.

~Repository: An interface for interacting with the database, implemented using Spring Data JPA.

Authentication and authorization management are implemented using Spring Security, with token-based authentication using Okta. Moreover, various features of Spring Boot are utilized to implement exception handling, logging, testing, and more.

Through this project, you can learn how to implement backend applications based on Spring and how to utilize various Spring technologies. Additionally, by implementing an application following the RealWorld specifications, it provides a basis for deciding which technology stack to choose through comparisons with various other technology stacks.

Spring Boot: Server side framework

JPA: Entity framework

Lombok: Provides automated getter/setters

Actuator: Application insights on the fly

Spring Security: Spring’s security

Devtools: Support Hot-Code Swapping with live browser reload

Okta: API authentication

H2: H2 database embedded version

• News and Notes
• Open-source implementation
• Extension Works and Ideas
• How to use
• Poster Talk and Slides
• FAQ
• References

• The code and related material in this main branch are for our ICML 2021 oral work, titled Conformal Prediction Interval for Dynamic Time-series (Xu et al. 2021a). Please use codes in this repository, as those downloaded from PMLR are not the most updated ones. You may direct any inquiries either to Chen Xu (cxu310@gatech.edu) or Yao Xie (yao.xie@isye.gatech.edu). The work is constantly updated to incorporate new feedback and ideas.
• We have significantly revised and extended the ICML 2021 work, which is accepted by 🌟IEEE Transactions on Pattern Analysis and Machine Intelligence🌟. The most recent version is also available on arxiv. The Journal_code branch contains updated codes, which essentially follow the same structure as those in this branch. Nevertheless, feel free to message us if you have any question regarding either branch.

• We are excited that the work has been integrated as a part of
  1. MAPIE, which is a scikit-learn-compatible module for predictive inference.
  2. Fortuna by Amazon AWS.
  3. PUNCC by the Artificial and Natural Intelligence Toulouse Institute.
  4. functime.ai for production-ready time series models.
  5. ConformalPrediction for Trustworthy AI in Julia.

• 🌟Sequential Predictive Conformal Inference for Time Series🌟 (Xu et al. 2023) is our latest work on this topic, where we advance EnbPI by considering time-adaptive re-estimation of residual quantiles, improving the performance of EnbPI and other recent CP methods.
• Conformal prediction set for time-series (Xu et al. 2022) focuses on time-series classification. It has been strongly accepted by the Distribution-free Uncertainty Quantification Workshop in ICML 2021 (DFUQ 2022), with the Poster here.
• Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data (Xu et al. 2021) adopts EnbPI for detecting anomalous traffic flows. Our method significantly outperforms competing methods (Table 1). It has been accepted by the Distribution-free Uncertainty Quantification Workshop in ICML 2021 (DFUQ 2021), with the Poster here.

• Required Dependency:
  • Basic modules: numpy, pandas, sklearn, scipy, matplotlib, seaborn.
  • Additional modules: statsmodels for implementing ARIMA, keras for building neural network and recurrent neural networks, and pyod for competing anomaly detection methods.
• General Info and Tests: This work reproduces all experiments in Conformal Prediction Interval for Dynamic Time-series (Xu et al. 2021a). In particular,
  • tests_paper.ipynb provides an illustration of how to generate the main figures (Figure 1-4) in the paper. The code contents are nearly identical to those in tests_paper.py.
  • tests_paper+supp.py reproduces all figures, including additional ones found in the Appendix. It is written in Jupyter notebook format, so that they are meant to be executed line by line.
• EnbPI implementation:
  • PI_class_EnbPI.py implements the class that contains EnbPI (line), Jackknife+-after-bootstrap (line, paper), Split/Inductive Conformal (line, paper), and Weighted Inductive Conformal (line, paper). We used ARIMA as another competing method.
  • Because conditional coverage (Figure 3) and anomaly detection (Figure 4) require problem-specific modifications, the code changes are not contained in “PI_class_EnbPI.py” but in their respective sections within tests_paper.py/tests_paper+supp.py.
• Other Function Files:
  • utils_EnbPI.py primarily contain plotting functions for all figures except Figure 4.
  • PI_class_ECAD.py implements ECAD based on EnbPI (see [Xu et al. 2021a, Section 8.5, Algorithm 2]) and utils_ECAD.py contains helpers for anomaly detection.
• Additional Files
  • The Data repository contains all dataset in our paperexcept the money laundry one for Figure 4 (due to size limit).
  • The Results repository is provided for your convenience to reproduce plots, since some experiments by neural network/recurrent neural networks can take some time to execute. It contains all .csv results files on all dataset.
• Broad Usage: To wrap EnbPI around other regression models and/or use on other data, one should:
  • If other regression models: Make sure the model has methods .fit(X_train, Y_train) to train a predictor and .predict(X_predict) to make predictions on new data. Most models in sklearn or deep learning models built by keras/pytorch are capable of doing so.
  • If other data: We have assumed that all our datasets are save as pandas.DataFrame and convertible to numpy.array. However, such assumptions are purely computational. Please feel free to adjust the data formate as long as it can be processed by regression models of choice.

• The poster for our work is available via this link.
• We are fortunate to pre-record a long presentation and give an oral presentation at the Proceedings of the 38th International Conference on Machine Learning (ICML 2021). The long presentation is available on Slideslive and the oral presentation will be given at the conference once the date is finalized.
• The slide for the talk is available here.

1. Encountering “NotImplementedError: Cannot convert a symbolic Tensor (lstm_2/strided_slice:0) to a numpy array” when using RNN as the regression model:

• See this github answer to resolve the problem, primarily due to numpy & python version issues.

• Xu, Chen and Yao Xie (2023). Sequential Predictive Conformal Inference for Time Series. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023
• Xu, Chen and Yao Xie (2023). Conformal prediction for time-series. Journal version, IEEE Transactions on Pattern Analysis and Machine Intelligence.
• Xu, Chen and Yao Xie (2021). Conformal prediction interval for dynamic time-series. Conference version, The Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021.
• Xu, Chen and Yao Xie (2022). Conformal prediction set for time-series. ICML 2022 Distribution-Free Uncertainty Quantification workshop.
• Xu, Chen and Yao Xie (2021). Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data. ICML 2021 Distribution-Free Uncertainty Quantification workshop.

This is a single board computer based on Motorola 68000. It features:

• 512kiB to 8MiB DRAM (SIMM 30 pins x2)
• basic memory protection / paging
• 8bits expansion bus with 4 slots
• PS/2 keyboard and mouse support
• serial link (TTL)
• boot with serial bootstrap or user flash

Some pictures here.

The block diagram below depicts the board architecture:

Most board glue logic is implemented with an Altera EPM7128S CPLD. This part is obsolete but modern replacement exists with Actel Microsemi Microchip ATF1508AS (never tested though). The CPLD implements:

• DRAM address multiplexing
• DRAM refresh
• basic memory management: base address translation and inbound check with power of two alignment constraints
• 8-bits expansion bus access, for 8-bits wide or 16-bits wide memory cycle from the 68000
• first stage boot procedure, by copying during reset the 512 first bytes from the flash memory (see MFP chip description below) at address 000000h in DRAM.

Expansion bus also uses a few discrete logic (74245 buffers and 74138 address decoder).

An additional 74148 encodes interrupt sources to CPU IPL0/1/2.

The CPLD provides very basic memory management capabilities, for user mode accesses only (FC2 set to 0). It relies on two value set with four registers. These registers are write only, 16-bits, but only the upper bit 15:8 are used.

Any user address is and’ed with MMU_MASK then or’ed with MMU_BASE. It results in a memory area allocated to a user program with a power of two size, with a base address aligned to its own size. Any user access that targets an address above the allocated size ends with a bus error.

The autovectored interrupt scheme is used (VPA# assertion). To save pins on the CPLD the FC0 and FC1 are not available to identify an interrupt acknowledge cycle, so any read cycle at FFFFFFh with FC2 set to 1 is considered as such.

This bus provides access to 5 peripherals:

• 4 available through dedicated connectors (2×10 female pin headers)
• 1 embeded to the board: the MFP

It provides for each peripheral up to 128kiB aperture into CPU memory space.

The table below lists the signals on this bus as seen for each slot.

This is how translates a byte access from the 68000 (UDS# or LDS# alone)

note: peripherals like the MFP ignore the address bits 16:1 and can ignore the ALE# signal.

This is how translates a 16-bits access from the 68000 (UDS# and LDS#)

Right after a reset, the CPLD starts by transfering 512 bytes of data from the MFP FLASH_DATA register into the first bytes of DRAM. Then the very first 68000 memory access (at 0000000h) is performed.

Because any software is to be much bigger than just 512-bytes, this chunk of code must itself include assembly instructions to copy the rest of the code – still accessing FLASH_DATA register.

The MFP contains two software images for the 68000:

• a serial bootstrap, loading from the serial interface the binary to jump in, that is enables by pressing ‘s’ key on a keyboard plugged into the PS/2 port
• or an embedded binary to jump in.

To change the embedded binary, the MFP firmware must be recompiled and programmed (the 68000 binary is a constant data table within the microcontroller firmware).

This is a software defined chip base on a PIC18F27K42 microcontroller. It provides:

• embedded flash with 2 software images for the 68000, selection by detecting pressed key at reset.
• IOs/timer with dedicated low-priority interrupt: two PS/2 port, one serial link, 1 kHz RTC timer
• 250 Hz system timer with dedicated highest-priority interrupt
• hard/soft reset
• debug register (POST or so)

The MFP has only two memory locations accessible through the expansion bus: one to set the actual register to access, the other to read or write data from/to the register.

This table below is no spec, it just shows the parts that equip the board on my desk.