MQ: Publish request files

ServiceStack Rabbit MQ
1

Hello!
There are files in my request, I need to place the request in the queue completely, but I can not do this.

When I do this as follows, everything works, but in the processing of the queue, null values:

using (var mqClient = MessageService.CreateMessageQueueClient()) {
    mqClient.Publish(Request.Files);
}

When I copy files to a stream, I get an exception:

public class QMsg {
    public Stream[] Files { get; set; }
 }
    
var filesMsg = new QMsg() { Files = new Stream[Request.Files.Length] };
    
for (int i = 0; i < Request.Files.Length; i++) {
    var ms = new MemoryStream();
     Request.Files[i].InputStream.CopyTo(ms);
    
     filesMsg.Files[i] = ms;
}
    
using (var mqClient = MessageService.CreateMessageQueueClient()) {
     mqClient.Publish(filesMsg);
}

As can be seen from the description of the exception, the CC tries to serialize the passed parameter to json and then to an array of bytes.

Question: How can I push files to a queue?

System.Reflection.TargetInvocationException
HResult=0x80131604
Message=Exception has been thrown by the target of an invocation.
Source=System.Private.CoreLib
StackTrace:
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at ServiceStack.PclExport.<>c__DisplayClass60_01.<CreateGetter>b__0(T o) at ServiceStack.Text.Common.WriteType2.WriteProperties(TextWriter writer, Object instance)
at ServiceStack.Text.Common.WriteType2.WriteLateboundProperties(TextWriter writer, Object value, Type valueType) at ServiceStack.Text.Common.WriteListsOfElements2.WriteGenericArrayMultiDimension(TextWriter writer, Array array, Int32 rank, Int32[] indices)
at ServiceStack.Text.Common.WriteListsOfElements2.WriteGenericArray(TextWriter writer, Array array) at ServiceStack.Text.Common.WriteType2.WriteProperties(TextWriter writer, Object instance)
at ServiceStack.Text.JsonSerializer.SerializeToString(Object value, Type type)
at ServiceStack.Text.JsonSerializer.SerializeToString[T](T value)
at ServiceStack.RabbitMq.RabbitMqProducer.Publish(String queueName, IMessage message, String exchange)
at ProductCatalog.Service.ProductCatalogServices.Post(CatalogAddImages request) in d:\Documents\TFSWorkspace\NewBackend\ProductCatalog\ProductCatalog.Service\ProductCatalogServices.cs:line 118
at ServiceStack.Host.ServiceRunner`1.d__13.MoveNext()

Inner Exception 1:
InvalidOperationException: Timeouts are not supported on this stream.

2

You shouldn’t, you’d either need to use a byte[] but I’d recommend instead sending a URL to the file and embed that in the message instead.

If you’re not using a file sharing Service (e.g. S3) you could save files in an RDBMS, but my preference would be to store files on the file system and maintain their metadata in an RDBMS.

3

The task is the following: the client sends pictures, the service responds that it has received pictures and puts them in the queue, then the processing pictures take the files from the queue, processes, unloads into s3(this is a custom s3 like storage), then puts the image IDs in the queue.

Do you offer a queue to implement in the database?

THX.

4

Again I’d recommend uploading any files to a file service immediately and give out the URL to anything that needs it, I’d be surprised is if this is not already the general strategy of dealing with files as it’s very inefficient and resource intensive to try and maintain files in App Servers memory.

e.g. for https://techstacks.io we upload all images to imgur immediately which has several advantages including allowing your App Servers (and all other Services) to be ephemeral.

1 Like